Thread by @eisokant

For my first tweet storm, I wanted to share a bit about the @sourcedtech open-source stack and how recent releases and announcements fit into the bigger picture.

In the coming years we believe #MLonCode will start to drastically change developer tooling in the following areas:

* Security & Compliance
* QA & Testing
* API Understanding
* Code Review/Quality

For #MLonCode to make an impact in these areas we believe the following ingredients are needed:

1. Large datasets of millions of repositories (thank you, @github)

To make this accessible to others, we released Public Git Archive:

External Tweet loading...
If nothing shows, it may have been deleted
by @sourcedtech view original on Twitter

2. A language-agnostic representation of code
* Universal Abstract Syntax Trees
* Abstractions for high-level concepts (functions, imports etc.)
* Ability to resolve cross-references

Which is why we're working on Babelfish, check it out here: dashboard.bblf.sh

3. Ability to query the history of millions of repositories, their source code, and the language agnostic representations of it

Which is why we're working on Gitbase as a SQL layer on top of Git: github.com/src-d/gitbase & github.com/src-d/go-git

4. Ability to learn from source code, which means fast scalable distributed processing of billions of Universal ASTs & their diffs

Which is why we're working on the engine (extending Apache Spark for #MLonCode): github.com/src-d/engine

5. An understanding of natural language in code since language is intent i.e. naturalness of code

Which is why we are training large-scale identifier embedding models on top of 10s of millions of repositories: github.com/src-d/models

6. An understanding of structure in code at UAST level, at the project level and at global dependency level

So far we have tackled structural embeddings on top of UAST's: github.com/src-d/models (combined with identifier embeddings, these are very powerful).

7. Shareable, versionable datasets & models for the community to be able to use and improve upon

Which is why we have github.com/src-d/modelfor… & github.com/src-d/datasets but also tools like code annotation: github.com/src-d/code-ann…

8. And the most important: an #MLonCode community of ML researchers, PL enthusiasts, dev. tooling engineers, data engineers, dev. advocates, PM's, designers and many other profiles that believe in the future of a language-agnostic ML powered development experience.

We believe these are the fundamental components for building #MLonCode applications (and here we'll have some exciting announcements as well coming up later this summer).