{{ value }}
@@ -24,11 +27,12 @@
- of -
+of
- tokens -
- +tokens
+- token -
+token
and
- {{ field }} -
+{{ field }}
- The DL4SE tool allows to easily create large-scale datasets that can be used to either run MSR studies or to - train DL models to automate SE tasks. Use our forms to define the characteristics of the dataset you would like - to build. The question mark icons in the forms will guide you by providing explanations about the different - filters you can use as well as processing operations that you can apply. + The DL4SE tool allows to easily create large-scale datasets that can be used to either run + MSR studies or to train DL models to automate SE tasks. Use our forms to define the + characteristics of the dataset you would like to build. The question mark icons in the forms + will guide you by providing explanations about the different filters you can use as well as + processing operations that you can apply.
- We continuously mine the source code of public GitHub repositories having at least 10 stars. While we - acknowledge that the number of stars is not a good proxy of a repository's quality or relevance, we found the - 10 stars threshold to be a reasonable mechanism to remove most of personal/toy projects unlikely to be relevant - for empirical studies. Note that, since DL4SE continuously updates the information about the mined projects, a - project excluded in the first place as having less than 10 stars can be mined a few days after once its number - of stars reaches the set threshold. Also, the source code of the mined repositories is kept in line with its - evolution on GitHub. This means that, pending a small time delay, you will always get fresh data as they appear - in the projects' latest snapshot on GitHub. + We continuously mine the source code of public GitHub repositories having at least 10 stars. + While we acknowledge that the number of stars is not a good proxy of a repository's quality + or relevance, we found the 10 stars threshold to be a reasonable mechanism to remove most of + personal/toy projects unlikely to be relevant for empirical studies. Note that, since DL4SE + continuously updates the information about the mined projects, a project excluded in the + first place as having less than 10 stars can be mined a few days after once its number of + stars reaches the set threshold. Also, the source code of the mined repositories is kept in + line with its evolution on GitHub. This means that, pending a small time delay, you will + always get fresh data as they appear in the projects' latest snapshot on GitHub.
- Besides providing the source code of the mined projects, we also implemented several filters and processing - steps frequently adopted in SE studies. For example, you can automatically remove from your dataset test code, - unparsable code, or code including non-ASCII characters. You can exclude exact and near duplicates from your - dataset, ask for an AST-based representation of each mined code instance, and much more! + Besides providing the source code of the mined projects, we also implemented several filters + and processing steps frequently adopted in SE studies. For example, you can automatically + remove from your dataset test code, unparsable code, or code including non-ASCII characters. + You can exclude exact and near duplicates from your dataset, ask for an AST-based + representation of each mined code instance, and much more!
- It depends on the size of the dataset and on the amount of requests running on our server. We allow for up to 10 - requests to run in parallel. If there are no runners available when you submit your request, it will be queued - until a runner is ready. You'll get an email as soon as your dataset is ready! In general, we do not expect the - creation of a dataset to take more than 24 hours. -
-- The dataset will be provided in JSON format. + It depends on the size of the dataset and on the amount of requests running on our server. + We allow for up to 10 requests to run in parallel. If there are no runners available when + you submit your request, it will be queued until a runner is ready. You'll get an email as + soon as your dataset is ready! In general, we do not expect the creation of a dataset to + take more than 24 hours.
+The dataset will be provided in JSON format.
- This is only our first step. In the next months we plan to integrate Python, C/C++, and Javascript. Also, we - will expand the types of datasets that can be built, providing forms specialized for the creation of datasets - to automate code summarization and code review. + This is only our first step. In the next months we plan to integrate Python, C/C++, and + Javascript. Also, we will expand the types of datasets that can be built, providing forms + specialized for the creation of datasets to automate code summarization and code review.
DL4SE is open source. Use the - issue tracker + + issue tracker + hosted on GitHub.
- You can build your dataset by considering as a single instance either an entire file or a function: + You can build your dataset by considering as a single instance either an entire file or a + function:
contentcontent_hashcharacterslinescode_tokenstotal_tokensis_testpath within the project's repository.
+ Whether the instance is a test file or a test case (depending on the granularity of
+ the dataset). We identify test files looking at their path within the
+ project's repository.
is_parsedfile instance was successfully parsed. If a file is not parsable, this means that we did not
- manage to extract the functions from it. Thus, we cannot have unparsable instances in the function-level dataset.
- For unparsable instances we do not compute specific information (e.g., the token counts).
+ Whether the file instance was successfully parsed. If a
+ file is not parsable, this means that we did not manage to extract the
+ functions from it. Thus, we cannot have unparsable instances in the
+ function-level dataset. For unparsable instances we do not compute specific
+ information (e.g., the token counts).
boilerplate_typefunction instance is one of following boilerplate types: constructor, getter,
- setter, builder, printer (i.e., toString), equality comparator (i.e., equals) and
- hash value computer (i.e., hashCode). The identification of these functions is done by analyzing their name.
+ Whether the function instance is one of following boilerplate types:
+ constructor, getter, setter, builder, printer (i.e., toString), equality
+ comparator (i.e., equals) and hash value computer (i.e.,
+ hashCode). The identification of these functions is done by analyzing
+ their name.
- Certain studies/models may benefit from information such as the AST-representation of source code, which is why we chose to offer
- the option of including an Abstract Syntax Tree (AST) representation of each exported instance in
- Extensible Markup Language (XML) format. The inclusion of this information within the dataset will increase its size drastically.
- Furthermore, since the AST is pre-computed for each instance at mining time, any processing applied to the source code implies that the syntax tree will be recomputed to be consistent with the code representation.
- For this reason, the inclusion of ASTs in the dataset may also increase the time needed to create the dataset.
+ Certain studies/models may benefit from information such as the AST-representation of source
+ code, which is why we chose to offer the option of including an Abstract Syntax Tree
+ (AST) representation of each exported instance in Extensible Markup Language
+ (XML) format. The inclusion of this information within the dataset will increase its size
+ drastically. Furthermore, since the AST is pre-computed for each instance at mining time,
+ any processing applied to the source code implies that the syntax tree will be recomputed to
+ be consistent with the code representation. For this reason, the inclusion of ASTs in the
+ dataset may also increase the time needed to create the dataset.
- Using these parameters, you can filter the projects (repositories) from which we will extract data. The first and
- only required criteria present in the form is the language. Specifying a value for this
- filter will not match projects whose main language matches the one specified, but rather all projects that have
- files written in that language. You can also filter repositories by specifying a lower bound for specific characteristics:
+ Using these parameters, you can filter the projects (repositories) from which we will
+ extract data. The first and only required criteria present in the form is the
+ language. Specifying a value for this filter will not match projects whose main
+ language matches the one specified, but rather all projects that have files written in that
+ language. You can also filter repositories by specifying a lower bound for specific
+ characteristics:
{{ filter }}
- We also provide options for excluding specific categories of projects from the selection, namely those that: + We also provide options for excluding specific categories of projects from the selection, + namely those that:
- Using these parameters, we filter the individual instances based on characteristics exhibited their the source - code. These include closed, half-closed or unbound ranges for the minimum and maximum number of: + Using these parameters, we filter the individual instances based on characteristics + exhibited their the source code. These include closed, half-closed or unbound ranges for the + minimum and maximum number of:
{{ filter }}
- Note that in the case of tokens, the column queried against depends on whether processing operations
- are applied during export. For example, removing comments from exported instances means that filtering will be
- performed according to the code_tokens, otherwise total_tokens is used. Apart from these
+ Note that in the case of tokens, the column queried against depends on whether
+ processing operations are applied during export. For example, removing comments from
+ exported instances means that filtering will be performed according to the
+ code_tokens, otherwise total_tokens is used. Apart from these
numeric attributes, you can also exclude all instances of:
- You can remove duplicated/similar entries from the dataset. This can be done by filtering out: + You can remove duplicated/similar entries from the dataset. This can be done by filtering + out:
content columns that may differ only in whitespace formatting.
+ Those with equivalent content columns that may differ only in whitespace
+ formatting.
content, posses the same ast
+ Those that in spite of small differences in content, posses the same
+ ast
structure.
- By definition, duplicates are a subset of near-clones, as identical contents imply identical AST structure. - These filters have a substantial computational cost. + By definition, duplicates are a subset of near-clones, as identical contents imply identical + AST structure. These filters have a substantial computational cost.
- This type of instance processing replaces a specified percentage of tokens with a user-specified token. - It works as follows: + This type of instance processing replaces a specified percentage of tokens with a + user-specified token. It works as follows:
content. Given that there are no guarantees as to the parsability of the masked code, we do not
- attempt to synchronize the AST to match.
+ All the tokens, replaced or otherwise, are joined back together to form the new masked
+ version of the original
+ content. Given that there are no guarantees as to the parsability of the
+ masked code, we do not attempt to synchronize the AST to match.
- Applies to each instance the abstraction process defined by Tufano et al. in their work An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation. - The user can specify the set of idioms to consider (i.e., tokens that should not be abstracted). + Applies to each instance the abstraction process defined by Tufano et al. in their work + + An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine + Translation . The user can specify the set of idioms to consider (i.e., tokens that should not be + abstracted).
- DL4SE is host to {{ formatNatural(count.funcs) }} functions, - sourced from {{ formatNatural(count.files) }} files, - originating from {{ formatNatural(count.repos) }} repositories. - In total, we have mined {{ formatBytes(size.code) }} of source code. - The platform currently has {{ count.users }} registered users, - and since its inception {{ count.tasks }} datasets have been constructed. - This amounts to roughly {{ formatBytes(size.tasks) }} in file size. + DL4SE is host to + {{ + formatNatural(count.funcs) + }} + functions, sourced from + {{ + formatNatural(count.files) + }} + files, originating from + {{ + formatNatural(count.repos) + }} + repositories. In total, we have mined + {{ formatBytes(size.code) }} of source code. The platform currently + has {{ count.users }} registered users, and since its inception + {{ count.tasks }} datasets have been constructed. This amounts to + roughly {{ formatBytes(size.tasks) }} in file size.
@@ -37,19 +47,13 @@