OSS | Democratizing Data

VanGohan PDF

Fri, 01 Sep 2023 18:00:00 -0700

digdaglog2sql

Thu, 05 May 2022 18:00:00 -0700

tabula-py

Sun, 01 Dec 2019 19:00:00 +0900

A recent update of tabula-py

Mon, 18 Feb 2019 01:26:00 +0900

Photo by [Joshua Rawson-Harris](https://unsplash.com/@joshrh19?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)

Photo by Joshua Rawson-Harris on Unsplash

This article is a repost of Patreon article published last December. I’m planning to bump up the next version of tabula-py within few weeks.

(Note: Oct 7th, 2019) As of Oct. 2019, I launched a documentation site and Google Colab notebook for tabula-py. The FAQ would be good place to execute accurate extraction.

This is my first post on patreon. Apologies for delayed announcement of recent update of tabula-py. I will introduce the key features of updates.

Use Tabula app template

Tabula app has template exporting feature to reuse same bounding box for extraction. tabula-py now load and extract with tabula app’s template.

dfs = tabula.read_pdf_with_template(
 './examples/data.pdf',
 './examples/data.tabula-template.json',
 pandas_options={'header': 0})

Support file-like object

Like many python libraries, tabula-py has been able to extract from file-like object.

# With file-like object 
pdf\_path = ‘tests/resources/data.pdf’
with open(pdf\_path, ‘rb’) as f:
 df = tabula.read_pdf(f)

# With pathlib 
from pathlib import Path
pdf_path = 'tests/resources/data.pdf'
df = tabula.read_pdf(Path(pdf_path))

Allow multiple area option

As of tabula-java v1.0.2, tabula can handle multiple area option.

pdf_path = 'tests/resources/MultiColumn.pdf'
# Relative area 
df_relative = tabula.read_pdf(
 pdf_path, pages=1,
 area=[[0, 0, 100, 50], [0, 50, 100, 100]], relative_area=True)

# Absolute area 
 df_absolute = tabula.read_pdf(
 pdf_path, pages=1, area=[[0, 0, 451, 212], [0, 212, 451, 425]])

Tip: Get table position

This is not a new feature, but I think it might be helpful for some PDFs.
Detailed post: https://github.com/chezou/tabula-py/issues/102

read_pdf with JSON contains position info, so you can get the table position as follows:

In [5]: tables = read_pdf("./examples/data.pdf", output_format="json", page=2)
In [9]: top = tables[0]['top']
In [10]: left = tables[0]['left']
In [11]: bottom = tables[0]['height'] + top
In [12]: right = tables[0]['width'] + left
In [13]: top, bottom, left, right
Out[13]: (0.0, 528.8800048828125, 0.0, 564.8800048828125)

If you have any question, ask on Stack Overflow!

Why OSS based machine learning is good?

Thu, 03 Aug 2017 12:56:59 +0900

This article is translation of Japanese version.

After releasing of TensorFlow, the movement of OSS-based machine learning is accelerating. François Chollet, the creator of Keras, says the essential point of this change. I think his phrase is enough, but in this article, I would like to organize why open source machine learning is great, and what recent trends are.

tl;dr

Machine learning and deep learning frameworks have become standard things for software engineers
Since arXiv becomes very famous, many papers are published before peer review of international conferences. This change made easier for other companies to validate the algorithm.
Many researchers have been started to study machine learning, machine learning researches in academia become Red Oceanic.
The strategy, “Make a great algorithm, but the implementation is secret” becomes a thing of the past.

Halcyon days

Five or ten years ago, almost all players working on advanced machine learning were in laboratories such as universities or large enterprises, or some advanced companies. In particular, the amount of data with a label was smaller than the present, and many researchers had been improving the performance by researching algorithms, by feature engineering.

Many researchers from academia studied state-of-the-art machine learning, posted to international conferences. Most of the insights were shared after peer review. Implementation was not shared as much as now, and each researcher had to reimplement the preceding research from scratch. A typical cycle for releasing new algorithms was a half year, in some cases more than a year.

There were few open source machine learning libraries/frameworks like Weka. scikit-learn, released in 2010, was not famous among software engineers. Many of us used libraries with single/few algorithms such as libsvm and liblinear.

Fast moving era

As of 2017, people who work in machine learning have significantly increased compared with 10 years ago. The center of machine learning has been moved from academia to companies with large data. In particular, software engineers, who have never worked on machine learning, entering deep learning world. I was surprised to hear that my friend of the community who had never worked on machine learning in business had started working on Deep Learning. The reasons for this movement are 1) it became general for companies to store large data that can be used for machine learning,
2) excellent machine learning frameworks have been increased, and 3) the GPU power leverage Deep Learning for efficient calculation.

Many open source libraries became popular not only in the frameworks of Deep Learning such as TensorFlow, Chainer, MXNet, Caffe 2, PyTorch but also by XGBoost, Lightgbm, which are famouse among kaggler. scikit-learn is also common tool as a framework to experiment with multiple algorithms.

The rise of “open papers”

This movement is supported by machine learning competition site “kaggle”, and by a place to post open papers called “arXiv”. (There is discussion arxiv does not have a peer review process and quality is not assured. So can we call the document as a research paper? But, in this post, I will call the research paper style report as “paper”)

The following article describes the number of paper submissions related to machine learning (especially Deep Learning) submitted to arXiv. According to this article, it is pointed out that the number of papers related to machine learning has more than quadrupled in 2017 compared to five years ago.

A Peek at Trends in Machine Learning
_Have you looked at Google Trends? It’s pretty cool — you enter some keywords and see how Google Searches of that term…_medium.com

Papers of arXiv are posted every day. It means, state-of-the-art results from such as Google, Facebook, Microsoft, etc. are published more and more before peer review. This is a challenge for the central laboratories of the traditional large enterprises to research and develop cutting edge algorithms of machine learning itself. Those companies usually set targets for a year or half a year. There is also criticism of “just adding parts”, but it is clear that the speed of developing machine learning algorithms is significantly fast.

In the field of machine translation, the breakthrough in deep learning was encoder-decoder and attention. The subsequent papers are not interesting, “I just put existing parts here.” I can’t understand why these papers come to the top conference.

Recently, for those who read new arXiv’s paper day and night, there is an system called “ariXiv Times” to better check new arrival documents.

arXivTimes Indicator
_Edit description_arxivtimes.herokuapp.com

Open papers accelerates Open source machine learning

This March, a paper about “Deep Forest” was published at arXiv, and it became a hot topic with the author claims that “performance is better than Deep Learning”.

\[1702.08835\] Deep Forest: Towards An Alternative to Deep Neural Networks
_Abstract: In this paper, we propose gcForest, a decision tree ensemble approach with performance highly competitive to…_arxiv.org

This method proposed in this paper, about one week (2017/3/5) after the publication (2017/2 / 28), R implementation came up and Python implementation came out after R one. A discussion was made with the following LightGBM issue on GitHub, and it came out that there was not reproducibility of the article, they can’t confirm the performance.

Support gcForest · Issue #331 · Microsoft/LightGBM
_LightGBM - A fast, distributed, high performance gradient boosting (GBDT, GBRT, GBM or MART) framework based on…_github.com

It is a symbolic event where the OSS implementation of the paper
appeared within a week after published in arXiv and the discussion of the community began.

I hear that it is increasing that the number of international conferences that require disclosing the implementation when a paper is submitted.

Conclusion

It is an essential task to develop the machine learning algorithm. Thanks to open papers, ML competition web site, and fast implementation of new algorithms as an OSS, we can adopt state-of-the-art knowledge into the business rapidly.

IMHO, it is becoming fun to focus on where we can make use of ML in business rather than developing the algorithm itself.

In other words, now, it is too hard to say “special machine learning algorithms that only our company can do”. Of course, people in academia will push these cutting-edge initiatives if they can prepare data. What is the evidence that one company invents a better algorithm quickly than most state-of-the-art people from tech giants like Google, Facebook, Microsoft, etc.? That is the reason for the strength of open source based machine learning.

Among academia, there is a famous phrase, “Standing on the shoulders of giants”, it means that we should thank previous research then we can go on to the next step. Even in machine learning based on open source, we can not ignore this phrase. We cannot ignore giants.

OSSベースの機械学習が強い理由

Thu, 03 Aug 2017 03:29:59 +0900

英語版はこちら_

TensorFlowの登場以降、OSSベースの機械学習の盛り上がりは加速しています。Kerasの作者のFrançois Cholletさんの言葉が、この状況を非常に端的に表しています。これだけでも十分だとは思いますが、この記事では、なぜオープンソースの機械学習が強いのか、最近のどういった流れがあるのかを整理したいと思います。

tl;dr

機械学習やDeep Learningのフレームワークが充実してきた
論文が査読前に公開され、他社も簡単にアルゴリズムの検証ができるようになった
多くのプレーヤーの参戦により、アカデミアでの機械学習の研究がレッドオーシャン化した
他社にないアルゴリズムで一発勝負、実装は秘密、というアプローチが厳しい

牧歌的な時代

5年前10年前の世界では、先端の機械学習に取り組んでいるのは大学などの研究室、大企業の研究所や一部の先進的な企業がほとんどでした。特に、ラベル付きのデータ量は現在よりも少なく、アルゴリズムを突き詰めて性能を向上させたり、特徴量エンジニアリングを頑張って性能を改善したりしていました。

この時代の先端の機械学習は、特に大学を中心としたアカデミアの人たちが国際会議に投稿をし、それが査読を通ったものが共有されるという形でした。実装が共有されることは今ほど多くはなく、各研究者は先行研究をスクラッチから再実装するということが多かったように思います。早くて半年、場合によっては1年以上のサイクルで新しいアルゴリズムがでてくるという時代でした。

オープンソースの機械学習ライブラリ/フレームワークとしては、古くはWekaのような機械学習用のライブラリを使う人もいれば、scikit-learnも2010年にリリースされましたが、今ほど多くのソフトウェアエンジニアの間ではメジャーではなかったように思います。どちらかというとlibsvmやliblinearなど単一/少数のアルゴリズムのものが多く使われていた時代でした。

変化の激しい時代

2017年の現在では、機械学習に取り組む人は10年前に比べれば圧倒的に増えたのはみなさんもご存知のとおりだと思います。アカデミアが中心だった時代から、データを持つ企業の人達にも広がってきました。特に、機械学習に取り組んだことのないソフトウェアエンジニアにも身近な存在になったと感じています。私の主催しているkawasaki.rbでも今まで業務で機械学習に取り組んだことのない人が、Deep Learningに取り組み始めたという話を聞いて驚きました。この背景には、やはり企業が機械学習に使えるデータを貯めることが一般的になったこと、そして優れた機械学習フレームワークが増えたこと、GPUによるDeep Learningの効率的な計算が可能となったことなどが挙げられます。

TensorFlow, Chainer, MXNet, Caffe2, PyTorchなどの数々のDeep Learningのフレームワークだけでなく、XGBoost, Lightgbmなどkaggleで流行って使われるようになったオープンソースのライブラリも多いです。当然、scikit-learnも複数のアルゴリズムを試すフレームワークとして当然使えるものになっています。

オープンな論文の盛り上がり

こうしたは流れを支えているのは、一つは機械学習のコンペティションサイトのkaggleと、もう一つはarxivと呼ばれるオープンな論文を投稿する場所の存在があります。（arxivは査読プロセスが入っていないため質が担保されておらず、それを論文と呼んでいいのか、などの議論はありますが、ここでは論文スタイルのドキュメントをざっくり論文と呼びます）

以下の記事に、arxivに投稿された機械学習関連（特にDeep Learning）の論文投稿数について記述があります。こちらの記事によれば、機械学習関連の論文の投稿数が、2017年には5年前に比べて4倍以上になっているという指摘がされています。

A Peek at Trends in Machine Learning

arxivの論文は、（査読がないこともあり）日々新しい論文が投稿されます。つまり、Google、Facebook、Microsoftなどの先端の取り組みが、査読前にどんどん論文として公開されています。これは、特に1年や半年単位で目標を設定して取り組むような従来型の大企業の中央研究所にとっては、機械学習の先端のアルゴリズム自体を研究開発するのはかなりのチャレンジになってきていると感じています。これについては「ただ部品を足しているだけ」という批判もありますが、牧歌的な時代のスピード感から目まぐるしく変わってきているのは明らかです。

最近では、日夜新しいarxivの論文を読み漁る人のために、新着論文をより良くチェックするためのariXiv Timesという取り組みもされています。

arXivTimesのダッシュボードをリリースしました

オープンな論文が加速するオープンソースの機械学習

今年の3月にDeep Forestという論文がarxivで公開され、”Deep Learningよりも性能が良い”という主張もあって話題になりました。日本でもQiitaに訳が公開されるなど話題になったので記憶に残っている人もいるかと思います。

実はこの論文で提案された手法は、論文が出た日（2017/2/28）の約一週間後（2017/3/5）にはR実装がでてきて、その後Python実装も出てきました。このやり取りは、以下のLightGBMのissueでなされ、論文の再現性がないのではないか（謳われているような性能が出なかった）という話になりました。

arxivで公開された論文のOSS実装が1週間後には実装が出てきてコミュニティの議論が始まる、というスピードの時代になっている象徴的な出来事です。

また、会議によっては論文投稿時に実装を公開することが求められる国際会議も増えてきていると聞きます。

雑感

機械学習のアルゴリズム自体を考えることは非常に重要な仕事です。そして、オープンな論文、コンペティション、そしてOSS化されるという流れの加速、そこから得られた新しい知見を素早くビジネスへ適用するのがとても速くできる時代になりました。

個人的には、アルゴリズムそのものよりも、それをビジネスに活かすにはどうすればいいか、というところにフォーカスすることのほうが楽しい時代になってきたのでは、とも思います。

この状況を翻って言うと、「自社だけができる特別な機械学習のアルゴリズム」というのを押し出していくのはうまくいかない時代に入ったともいえます。（もちろん、アカデミアの人たちはデータを用意できればこうした最先端の取り組みを今後も推し進めていくでしょうが）企業1社で、Google、Facebook、Microsoftなど世界の多くの最先端の人たちよりも素晴らしいアルゴリズムが素早く出せる理由は何でしょうか？それが、オープンソースベースの機械学習の強い理由でしょう。

アカデミアでは昔から「巨人の肩に乗る」という言葉が知られていますが、先行研究という巨人に乗って次に進むということを示しています。オープンソースベースの機械学習でも、「巨人の肩に乗る」ということが顕著になってきています。

tabula-py now able to extract remote PDF and multiple tables at once

Sun, 28 May 2017 11:18:39 +0900

(Note: Oct 7th, 2019) As of Oct. 2019, I launched a documentation site and Google Colab notebook for tabula-py. The FAQ would be good place to execute accurate extraction.

tabula-py is a Python library which enables you to extract tables from PDF into pandas DataFrames. Today, I released v0.8.0. In this post, I will introduce improvements after previous post of tabula-py. If you don’t familiar with tabula-py, you can see previous one.

Change Notes

Able to read remote PDF passing URL
\[Experimental\] Add multiple_tables mode
Add batch conversion method:convert_into_by_batch()
Add encoding option
Add java_options
Will deprecate read_pdf_table() method

I will explain important features.

Read remote PDF passing URL

If you want extract a DataFrame from the internet, you can extract remote PDF without downloading it manually.

read_pdf("https://github.com/tabulapdf/tabula-java/raw/master/src/test/resources/technology/tabula/12s0324.pdf")

\[Experimental\] Add “`multiple_tables"` mode

tabula-py is a simple wrapper of tabula-java, it was hard to handle multiple tables in a page. But now, you can extract multiple tables in a page using multiple_tables option.

read_pdf('tests/resources/data.pdf', pages=2, multiple_tables=True)

This function create a list of DataFrames via JSON from tabula-java, so if tabula-java’s JSON format will change, the output could be broken. If you see CParserError , try to set multiple_tables option.

Add batch conversion method: “`convert_into_by_batch()"`

After tabula-java v0.9.2, we can extract tables from PDF by batch. You can use this function through convert_into_by_batch() method.

convert_into_by_batch(path_to_dir, output_format='csv')

You should set directory path of PDFs, not the specific pdf path.

tabula-py extracts tables same directory as input files.

TODOs

There are several problems those may be fixed after releasing of tabula-java 0.9.3. e.g) Handling embedded font, including Japanese…

Waiting for your collaboration!

If you have any troubles with tabula-py, please file an issue on GitHub. I don’t want to receive emails because the answer will not share to other people. Make sure fill the issue template, it will reduce many costs for me to solve the problem.

tabula-py: Extract table from PDF into Python DataFrame

Mon, 09 Jan 2017 14:09:08 +0900

(Oct 7th, 2019) As of Oct. 2019, I launched a documentation site and Google Colab notebook for tabula-py. The FAQ would be good place to execute accurate extraction.

Screenshots in this article is based on the old version interface. See the latest version example in the Colab notebook.

Today, I released tabula-py 0.3.0, which extracts table from PDF into Python pandas’s DataFrame.

It is simple wrapper of tabula-java and it enables you to extract table into DataFrame or JSON with Python. You also can extract tables from PDF into CSV, TSV or JSON file.

tabula is a tool to extract tables from PDFs. It is GUI based software, but tabula-java is a tool based on CUI. Though there were Ruby, R, and Node.js bindings of tabula-java, before tabula-py there isn’t any Python binding of it. I believe PyData is a great ecosystem for data analysis and that’s why I created tabula-py. If you are familiar with R, I highly recommend to use tabulizer, which has the most richest bindings including rich GUI.

You can install tabula-py via pip:

pip install tabula-py

With tabula-py, you can get DataFrame with read_pdf() method.

example of read_pdf()

example of read_pdf()

You can also extract tables as JSON format:

example of JSON

You can extract tables into a file like JSON, CSV or TSV with convert_into() method.

You can see more examples in Jupyter notebook.

I hope you will enjoy data wrangling with tabula-py. Any feedback would be welcome!

Waiting for your collaboration!

If you have any trouble with tabula-py, please file an issue on GitHub. I don’t want to receive emails because the answer will not share with other people. Make sure to fill the issue template, it will reduce many costs for me to solve the problem. Or, I also check StackOverflow. You can ask about it.

OSS | Democratizing Data

VanGohan PDF

digdaglog2sql

tabula-py

A recent update of tabula-py

Photo by [Joshua Rawson-Harris](https://unsplash.com/@joshrh19?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)

Use Tabula app template

Support file-like object

Allow multiple area option

Tip: Get table position

Other tabula-py articles

Why OSS based machine learning is good?

tl;dr

Halcyon days

Fast moving era

The rise of “open papers”

Open papers accelerates Open source machine learning

Conclusion

OSSベースの機械学習が強い理由

tl;dr

牧歌的な時代

変化の激しい時代

オープンな論文の盛り上がり

オープンな論文が加速するオープンソースの機械学習

雑感

tabula-py now able to extract remote PDF and multiple tables at once

Change Notes

Read remote PDF passing URL

\[Experimental\] Add “`multiple_tables"` mode

Add batch conversion method: “`convert_into_by_batch()"`

TODOs

Waiting for your collaboration!

Other tabula-py articles

tabula-py: Extract table from PDF into Python DataFrame

example of read_pdf()

example of JSON

Waiting for your collaboration!

Other tabula-py articles

OSS | Democratizing Data

VanGohan PDF

digdaglog2sql

tabula-py

A recent update of tabula-py

Photo by [Joshua Rawson-Harris](https://unsplash.com/@joshrh19?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)

Use Tabula app template

Support file-like object

Allow multiple area option

Tip: Get table position

Other tabula-py articles

Why OSS based machine learning is good?

tl;dr

Halcyon days

Fast moving era

The rise of “open papers”

Open papers accelerates Open source machine learning

Conclusion

OSSベースの機械学習が強い理由

tl;dr

牧歌的な時代

変化の激しい時代

オープンな論文の盛り上がり

オープンな論文が加速するオープンソースの機械学習

雑感

tabula-py now able to extract remote PDF and multiple tables at once

Change Notes

Read remote PDF passing URL

\[Experimental\] Add “multiple_tables" mode

Add batch conversion method: “convert_into_by_batch()"

TODOs

Waiting for your collaboration!

Other tabula-py articles

tabula-py: Extract table from PDF into Python DataFrame

example of read_pdf()

example of JSON

Waiting for your collaboration!

Other tabula-py articles

\[Experimental\] Add “`multiple_tables"` mode

Add batch conversion method: “`convert_into_by_batch()"`