infer-old-pldi-overview.tex

Now that we have introduced the problem,
we can flesh out our philosophy and overall
approach. To concretize our discussion,
\figref{infer:fig:cljs} demonstrates our tool's output
when generating types for the 1,776 line file cljs.compiler.

Its main function is \clj{emit}, which
effectfully converts a map-based AST
to JavaScript.
The AST is created by functions in cljs.analyzer,
a significantly larger 4,366 line Clojure file.
Without inspecting cljs.analyzer,
our tool annotates \clj{emit} on line \ref{infer:listing:cljs:emit}
with a recursive AST type \clj{Op} (lines \ref{infer:listing:cljs:Op}-\ref{infer:listing:cljs:Op-End}).

%
%Similar to our opening example, it uses the \clj{:op}
%key to disambiguate between (16) cases, and has recursive
%references (\clj{Op}).

%We just present the first 4 cases.
%The first case \clj{':binding} has 4 required
%and 8 optional entries.
%The \clj{:info} and \clj{:env} entries refer to
%other \clj{HMap} type aliases generated by the tool.
%Similar to \clj{:op},
%the \clj{:local} entry maps to a keyword singleton
%type,
%however our tool wisely chose to cluster types 
%based on the \clj{:op} entry since it is common to all cases.


%\Dsection{Philosophy}

An important question to address is ``how accurate are these annotations?''.
Unlike previous work in this area~\infercitep{An10dynamicinference}, we do not aim for soundness guarantees
in our generated types. 
A significant contribution of our work is a tool that Clojure programmers
can use to help learn about and specify their programs.
In that spirit, we strive to generate annotations meeting more qualitative criteria.
Each guideline by itself helps generate more useful annotations and,
as we discuss in \secref{infer:sec:experiment1},
they combine in interesting ways help to make up for shortcomings.
%in generated annotations.
%which we outline along with a commentary
%judging \figref{infer:fig:cljs} along these lines.

\paragraph{Choose recognizable names}
%Typed Clojure and clojure.spec annotations are abundant
%with useful names for types.
Assigning a good name for a type increases
readability by succinctly conveying its purpose.
Along those lines, a good name for the AST representation
on lines \ref{infer:listing:cljs:Op}-\ref{infer:listing:cljs:Op-End}
might be \clj{AST} or \clj{Expr}.
However, these kinds of names can be very misleading when incorrect, so
instead of guessing them,
our tool takes a more consistent approach and generates \emph{easily recognizable}
names based on the type the name points to.
Then, those with a passing familiarity with the data flowing through the program
can quickly identify and rename them.
For example,
\begin{itemize}
  \item
    \clj{Op} (lines \ref{infer:listing:cljs:Op}-\ref{infer:listing:cljs:Op-End})
    is chosen because \clj{:op} is
    clearly the dispatch key (the \clj{:op} entry is also helpfully placed
    as the first entry in each case to aid discoverability),
  \item
    \clj{ColumnLineContextMap} (lines \ref{infer:listing:cljs:ColumnLineContextMap}-\ref{infer:listing:cljs:ColumnLineContextMapEnd})
    enumerates the keys of the map type it points to,
  \item
    \clj{NameShadowMap} and \clj{FnScopeFnSelfNameNsMap} (%referenced on
    line
    \ref{infer:listing:cljs:Op:op:binding:NameShadowMap}% and \ref{infer:listing:cljs:Op:op:binding:FnScopeFnSelfNameNsMap}
    )
    similarly, and
  \item
    \clj{HMap49305} (lines \ref{infer:listing:cljs:HMap49305}-\ref{infer:listing:cljs:HMap49305End})
    shows how our tool fails to give names to certain combinations
    of types (we discuss the severity of this particular situation in
    \secref{infer:sec:experiment1}).
\end{itemize}

\begin{figure}
  % indented so line numbers can line up more tastefully
\begin{cljlistingnumbered}
  (defalias Op(*@\label{infer:listing:cljs:Op}@*) ; omitted some entries and 11 cases
    (U (HMap :mandatory(*@\label{infer:listing:cljs:Op:op:bindingStart}@*)
             {:op ':binding,(*@\label{infer:listing:cljs:Op:op:binding}@*) :info (U NameShadowMap(*@\label{infer:listing:cljs:Op:op:binding:NameShadowMap}@*) FnScopeFnSelfNameNsMap(*@\label{infer:listing:cljs:Op:op:binding:FnScopeFnSelfNameNsMap}@*)), ...}
             :optional(*@\label{infer:listing:cljs:Op:optional}@*)
             {:env ColumnLineContextMap, :init Op,(*@\label{infer:listing:cljs:Op:optional:init:Op}@*) :shadow (U nil Op),(*@\label{infer:listing:cljs:Op:optional:shadow:Op}@*) ...})(*@\label{infer:listing:cljs:Op:optionalEnd}@*)(*@\label{infer:listing:cljs:Op:op:bindingEnd}@*)
      '{:op ':const,(*@\label{infer:listing:cljs:Op:op:const}@*) :env HMap49305,(*@\label{infer:listing:cljs:Op:op:const:HMap49305}@*) ...}
      '{:op ':do,(*@\label{infer:listing:cljs:Op:op:do}@*) :env HMap49305,(*@\label{infer:listing:cljs:Op:op:do:HMap49305}@*) :ret Op,(*@\label{infer:listing:cljs:Op:op:do:Op}@*) :statements (Vec Nothing)(*@\label{infer:listing:cljs:Op:op:do:statements}@*), ...}
      ...))(*@\label{infer:listing:cljs:Op-End}@*)
  (defalias ColumnLineContextMap(*@\label{infer:listing:cljs:ColumnLineContextMap}@*)
    (HMap :mandatory {:column Int, :line Int} :optional {:context ':expr}(*@\label{infer:listing:cljs:ColumnLineContextMap:optional}@*)))(*@\label{infer:listing:cljs:ColumnLineContextMapEnd}@*)
  (defalias HMap49305 ; omitted some extries(*@\label{infer:listing:cljs:HMap49305}@*)
    (U nil
       '{:context ':statement, :column Int, ...} '{:context ':return, :column Int, ...}
       (HMap :mandatory {:context ':expr, :column Int, ...} :optional {...})))(*@\label{infer:listing:cljs:HMap49305End}@*)
  (ann emit [Op -> nil])(*@\label{infer:listing:cljs:emit}@*)
  (ann emit-dot [Op -> nil])(*@\label{infer:listing:cljs:emit-dot}@*)
\end{cljlistingnumbered}
\caption{
         Sample raw output from our
         tool inferring types for cljs.compiler,
         a 1,776 line file defining the code generation phase of
         a production-quality compiler.
         %While imperfect, the recursive type \clj{Op} generated by our tool 
         %is an invaluable starting
         %point for further annotations.
         %It describes the AST format for a compiler called cljs.compiler
         %(\secref{infer:chap:evaluation})
         %, and
         Its AST format is inferred as \clj{Op} (lines \ref{infer:listing:cljs:Op}-\ref{infer:listing:cljs:Op-End})
         with 22 recursive references
         (like lines \ref{infer:listing:cljs:Op:optional:init:Op}, \ref{infer:listing:cljs:Op:optional:shadow:Op}, \ref{infer:listing:cljs:Op:op:do:Op})
         and 14 cases distinguished by \clj{:op} (like lines \ref{infer:listing:cljs:Op:op:binding},
         \ref{infer:listing:cljs:Op:op:const}, \ref{infer:listing:cljs:Op:op:do}),
         5 of which have optional entries (like lines \ref{infer:listing:cljs:Op:optional}-\ref{infer:listing:cljs:Op:optionalEnd}).
         To improve inference time,
         only the code emission unit tests were exercised (299 lines containing 39 assertions)
         which normally take 40 seconds to run, from which we
         generated 448 lines of types and 517 lines of specs
         in 2.5 minutes on a 2011 MacBook Pro (16GB RAM, 2.4GHz i5).
}
\label{infer:fig:cljs}
  %(ann emit-let [Op Any -> Any])(*@\label{infer:listing:cljs:emit-let}@*)
%    '{:op ':fn-method,
%      :body Op,
%      :children '[':params ':body],
%      :env HMap49305,
%      :fixed-arity Int,
%      :form (Coll (Coll Any)),
%      :name Op,
%      :params '[Op],
%      :recurs nil,
%      :type nil,
%      :variadic? false}
%    '{:op ':host-call,
%      :args '[Op],
%      :children Any,
%      :env context-statement-tmp-HMap-alias20275,
%      :form (Coll Sym),
%      :method Sym,
%      :tag Any,
%      :target Op}
%    '{:op ':host-field,
%      :children '[':target],
%      :env context-statement-tmp-HMap-alias20275,
%      :field Sym,
%      :form (Coll Sym),
%      :tag Sym,
%      :target Op}
%    '{:op ':if,
%      :children '[':test ':then ':else],
%      :else Op,
%      :env context-statement-tmp-HMap-alias20275,
%      :form (Coll Any),
%      :tag (Set (U nil Sym)),
%      :test Op,
%      :then Op,
%      :unchecked Boolean}
%    '{:op ':invoke,
%      :args '[Op],
%      :children '[':fn ':args],
%      :env context-statement-tmp-HMap-alias20275,
%      :fn Op,
%      :form (Coll Any),
%      :tag Sym}
%    (HMap
%      :mandatory
%      {:op ':js,
%       :env context-statement-tmp-HMap-alias20275,
%       :form (Coll (U nil Str Sym)),
%       :js-op Sym,
%       :numeric nil,
%       :tag Sym}
%      :optional
%      {:args '[Op Op],
%       :children '[':args],
%       :code Str,
%       :segs (Coll Str)})
%    (HMap
%      :mandatory
%      {:op ':js-var, :name Sym, :ns Sym}
%      :optional
%      {:tag Sym})
%    '{:op ':let,
%      :bindings '[Op Op Any],
%      :body Any,
%      :children Any,
%      :env context-statement-tmp-HMap-alias20275,
%      :form Any,
%      :tag Any}
%    (HMap
%      :mandatory
%      {:op ':local,
%       :env context-statement-tmp-HMap-alias20275,
%       :form Sym,
%       :info Op,
%       :local (U ':arg ':let),
%       :name Sym}
%      :optional
%      {:arg-id Int, :init Op, :tag Sym})
%    '{:op ':map,
%      :children '[':keys ':vals],
%      :env context-statement-tmp-HMap-alias20275,
%      :form AMap,
%      :keys '[Op],
%      :tag Sym,
%      :vals '[Op]}
%    (HMap
%      :mandatory
%      {:op ':var, :name Sym, :ns Sym}
%      :optional
%      {:arglists (Coll Any),
%       :arglists-meta (Coll nil),
%       :column Int,
%       :doc Str,
%       :end-column Int,
%       :end-line Int,
%       :env context-statement-tmp-HMap-alias20275,
%       :file (U nil Str),
%       :fn-var Boolean,
%       :form Sym,
%       :info (U nil ColumnFileLineMap),
%       :line Int,
%       :max-fixed-arity Int,
%       :meta
%       (U
%         ColumnFileLineMap__0
%         FileArglistsColumnMap
%         ColumnEndColumnEndLineMap),
%       :method-params (Coll (Coll Sym)),
%       :protocol-impl nil,
%       :protocol-inline nil,
%       :ret-tag Sym,
%       :tag Sym,
%       :top-fn ArglistsArglistsMetaMaxFixedArityMap,
%       :variadic? Boolean})))
\end{figure}


%Good names can sometimes be reconstructed from the program source,
%like function or parameter names, and other times 
%we can use the shape of a type to summarize it.

\paragraph{Favor compact annotations}
Literally translating runtime observations into
annotations without compacting them
leads to unmaintainable and impractical types resembling
TypeWiz's annotation for \clj{nodes} (\figref{fig:infer:typewiz}).
To avoid this, we
  use optional keys where possible, like line \ref{infer:listing:cljs:ColumnLineContextMap:optional},
  infer recursive types like \clj{Op}, and
  reuse type aliases in function annotations, like
    \clj{emit} and \clj{emit-dot} (lines \ref{infer:listing:cljs:emit}, \ref{infer:listing:cljs:emit-dot}).
These processes of compacting annotations often makes them more general,
which leads into our next goal.

%Idiomatic Clojure code rarely mixes certain types in the same position,
%unless the program is polymorphic. Using this knowledge---which we observed
%by the annotations and specs assigned to idiomatic Clojure 
%code---we can rule out certain combinations of types to compact our
%resulting output, without losing information that would help us
%type check our programs.

\paragraph{Don't overspecify types}
Poor test coverage can easily skew the results of dynamic analysis tools,
so we choose to err on the side of generalizing types
where possible.
Our opening example \clj{nodes} (\figref{fig:infer:nodes})
is a good example of this---our inferred type (\figref{fig:infer:nodestype})
is recursive, despite \clj{nodes} only being tested with a tree of height 2.
This has several benefits.
\begin{itemize}
  \item We avoid exhausting the pool of easily recognizable names
    by generalizing types to communicate the general role
    of an argument or return position.
    For example, \clj{emit-dot} (line \ref{infer:listing:cljs:emit-dot})
    is annotated to take \clj{Op}, but in reality accepts only a subset
    of \clj{Op}.
    Programmers can combine the recognizability of \clj{Op} with the
    suggestive name of \clj{emit-dot} (the dot operator in Clojure handles host interoperability) to decide whether, for instance,
    to split \clj{Op} into smaller type aliases
    or add type casts in the definition of \clj{emit-dot} to please 
    the type checker
    (some libraries require more casts than others to type check, as discussed in \secref{infer:sec:experiment2}).
  \item Generated Clojure spec annotations (an extension discussed in \secref{infer:sec:spec-extension})
        are more likely to accept valid input with specs enabled, even with incomplete unit tests
        (we enable generated specs on several libraries in \secref{infer:sec:experiment3}).
  \item Our approach becomes more amenable to extensions improving the running time
        of runtime observation without significantly deteriorating annotation quality,
        like lazy tracking (\secref{infer:sec:lazy-tracking}).
\end{itemize}
%For example,
% - helps contract checking
% - can simply add a type cast to please type checker
%   - particularly cheap due to occurrence typing here (assert (= :dot (:op ast)))
% - avoids large types

% this is really an approach
%\paragraph{Recursive}
%Maps in Clojure are often heterogeneous, and recursively defined.
%Typed Clojure and clojure.spec supplies mechanisms for the most
%common case: maps of known keyword entries.
%We strategically \textbf{squash} flat types to be recursive
%based on their unrolled shape.
%For example, a recursively defined union of maps almost always
%contains a known keyword ``tag'' mapped to a keyword.
%By identifying this tag, we can reconstruct a good recursive
%approximation of this type.

%\Dsection{Naming}
%
%For a type to be immediately useful to a programmer, it helps
%to have a great name. We explored several avenues for
%generating good names.
%
%For types that occured as function arguments, the name of
%the argument often indicated its role in the program.
%Names like \clj{config} or \clj{env} are often used
%for an environment being functionally threaded through
%the program.
%
%Similarly, types that occur as values in configuration
%maps often have descriptive keys.
%For example, one of our case studies, a Star Trek 
%game written in Clojure,
%features a configuration map with a \clj{:stardate}
%entry containing
%a map that of three number entries: 
%\clj{:start},
%\clj{:current}, and
%\clj{:end}.
%
%What if a type occurs in the return position of a function?
%Sometimes these are named by \textbf{let} binding the result
%of the computation.
%
%Failing these heuristics, we fall back of several approaches
%to naming.
%First, if we are naming a keyword map which is part of a tagged
%union, we use the tag as the name. For example, if the tagged entry
%maps \textbf{:op} to \textbf{:fn}, we name this map \textbf{FnOp}.
%Otherwise, for maps with less than three entries, we simply
%enumerate its entries as the name.
%Finally, for large keyword maps, we give an abbreviation
%of its keyset as a name.

%\Dsection{Approach}

%
Our general approach to generating types
is separated into two phases---\textbf{collection} and
\textbf{inference}.
%
The collection phase (\secref{infer:sec:formal:collection-phase}),
gathers observations about a running program.
This is achieved by instrumenting the program and exercising
it, usually by running its unit tests,
with space-efficient tracking (\secref{infer:sec:space-efficient-tracking})
avoiding redundant traversals of values.
%
The inference phase (\secref{infer:sec:formal:inference-phase})
uses these runtime observations to generate the final type annotations,
with recursive types, optional entries, and good names.

%is split into 
%\textbf{instrumentation}, involves
%rewriting the code we wish to annotate such
%that we can record its runtime behavior.
%In this phase, we require the programmer to
%indicate which code we wish to generate types
%for, in advance.
%
%Once instrumented, we observe our running program
%via \textbf{runtime tracking}. To exercise our programs,
%we usually run their unit tests, generative tests,
%or just normally run the program (eg. to generate types for
%a game, we can simply play the game for a few minutes).
%We accumulate the results of tracking via \textbf{paths}.
%If we think of types as trees and supply a label
%for each branching path, our inference results
%specify the type down a particular path in this tree.
%Both phases are described in ,
%collectively as the \emph{collection phase}.
%
%Finally, the information collected during runtime tracking
%is combined into annotations by our \textbf{inference algorithm}.
%We first combine all inference result into a large tree of
%types. If we were to convert this tree into annotations directly,
%our annotations would be too specific---they would be too
%deep and fine-grained.
%Instead, our algorithm iterates over several passes to massage
%this tree, generating good names for the nodes, compacting similar
%types across the tree, and
%eventually converting the tree into a directed graph by reconstructing
%recursive types.
%This \emph{inference phase} is described in 
%\secref{infer:sec:formal:inference-phase}.

%\Dsection{Algorithm}

The first pass in the inference phase generates a naive type environment
from runtime observations (described in \secref{infer:sec:formal:inference-phase:genenv}). Types are fully unrolled, appearing
similar to TypeWiz's final annotation for \clj{nodes}
(\figref{fig:infer:typewiz})---except it is the starting point of
our algorithm.

A key hypothesis in our algorithm is that functions in the same
file operate on related data. Our inference is built
to be used per-file and aggressively merges all apparently-related HMap
types---firstly types immediately nested within each other (\secref{infer:sec:formal:inference-phase:squash-local}),
and then across type aliases
(\secref{infer:sec:formal:inference-phase:squash-global}).
This often yields useful types in our benchmarks.
This approach has limited success for libraries providing polymorphic
functions, since unit tests may use sample data that are unrelated
to the details of a function.
Our algorithm excels
with programs that mainly operate on one or two (possibly recursive) map-based
data representations,
which, outside of polymorphic libraries, are common in the Clojure ecosystem
in our experience.

%\begin{Verbatim}
%(defn f [x] (inc x))
%\end{Verbatim}
%
%\begin{figure}
%\begin{cljlisting}
%(defn vertices [m]
%  (case (:op m)
%    :leaf 1
%    :node (+ 1 (:left m)
%               (:right m))))
%\end{cljlisting}
%\label{code:vertices}
%\end{figure}

%%%% An old introductory example %%%%%%

%Consider the problem of inferring the type for
%the following Clojure program, a one
%argument function $f$ that increments numbers.
%
%\begin{verbatim}
%(defn f [x] (inc x))
%\end{verbatim}
%
%We might unit test this feature on the integers,
%to ensure incrementing $1$ gives $2$.
%
%\begin{verbatim}
%(deftest f-test
%  (is (= (f 1) 2)))
%\end{verbatim}
%
%We can instrument this program to observe its runtime behavior.
%
%\begin{verbatim}
%(deftest f-test
%  (is (= ((track f ['f]) 1) 2)))
%\end{verbatim}
%
%The $track$ function takes a value and a \emph{path},
%which tracks the subcomponent of the current environment
%the given value represents. For example, the path
%
%\begin{verbatim}
%  ['f :domain]
%\end{verbatim}
%
%represents the domain of $f$.
%
%After instrumentation we collect some inference results,
%associating paths with types: $path : \tau$.
%
%\begin{verbatim}
%  [['f :domain] Int]
%  [['f :range] Int]
%\end{verbatim}
%
%We combine this information into a type environment
%$\Gamma$ mapping variables to types: $\{x : \tau\}$.
%
%\begin{verbatim}
%  {x : [Int -> Int]}
%\end{verbatim}
%
%Now consider the case where we add a new unit test.
%
%\begin{verbatim}
%(deftest f-test
%  (is (= (f 1) 2))
%  (is (= (f 2.5) 3.5)))
%\end{verbatim}
%
%We now have a new set of inference results:
%
%\begin{verbatim}
%  [['f :domain] Num]
%  [['f :range] Num]
%\end{verbatim}
%
%which we want to combine with our previously inferred type environment
%
%\begin{verbatim}
%  {x : [Int -> Int]}
%\end{verbatim}
%
%How do join $[Int -> Int]$
%and $[Num -> Num]$?
%We have several options.
%
%
%\begin{verbatim}
%[(I Num Int) -> (U Num Int)]
%\end{verbatim}
%
%\begin{verbatim}
%[(U Num Int) -> (U Num Int)]
%\end{verbatim}
%
%\begin{verbatim}
%(IFn [Int -> Int]
%     [Num -> Num])
%\end{verbatim}

%\begin{figure}
%\begin{tikzpicture}[level distance=0.8cm, scale=0.9]
%\tikzstyle{every node}=[font=\small]
%\tikzset{grow'=down}
%\tikzset{every tree node/.style={align=center,anchor=north}}
%\Tree [.\node[draw](P0){\texttt{P0 = (Pairof P1 P2)}};
%        [.\node[draw](P2){\texttt{P2 = (Pairof P3 N)}};
%          [ \texttt{N} ]
%          [.\node[draw](P3){\texttt{P3 = (Pairof N N)}};
%            [ \texttt{N} ] [ \texttt{N} ]]
%]
%        [.\node[fill=gray!80](P1){\texttt{P1 = (Pairof N N)}};
%          [ \texttt{N} ] [ \texttt{N} ] ]
% ]
%\end{tikzpicture}
%%
%%\hspace{0.15cm}
%%
%\begin{tikzpicture}[level distance=0.8cm]
%\tikzstyle{every node}=[font=\small]
%\tikzset{grow'=down}
%\tikzset{every tree node/.style={align=center,anchor=north}}
%\Tree [.\node[fill=gray!80](P0){\texttt{P0 = (Pairof ($\cup$ N P0) ($\cup$ N P2))}};
%        [.\node[draw](P2){\texttt{P2 = (Pairof P3 N)}};
%          [ \texttt{N} ]
%          [.\node[draw]{\texttt{P3 = (Pairof N N)}};
%            [ \texttt{N} ] [ \texttt{N} ]]
%]
%          [ \texttt{N} ] [ \texttt{N} ] 
% ]
%%\draw[semithick,->] (P1)..controls +(west:1) and +(west:1)..(P0);
%\end{tikzpicture}
%\end{figure}
%