hssqlppp-0.1.0: docs/TODO
= new immediate todos:
get typing working for wrapper generator demo -> release
then in some order:
- docs and examples - some basics
- roundtrip + report
start with users sql: either source files, a database or mixture
say if parse ok, -> source files or pg_dumps
list type errors from typechecking asts
pretty print the catalog from sql source or dumps from typechecking
check this catalog against pg catalog
also, rename environment type at least
- nullability plus support in wrapper generator
then maybe:
full api review, renames, error handling work, plus sqltypes review
= milestones
not in any particular order
== type check for implicit join: need brain to work on
== use better database config
command line args and config file for db, username and password, and
add prompt style so works like psql
== separate out documentation program to separate package
-> how to do plugins for extra commands?
== documentation work
reorganise doc files (changelog, development, questions, README, TODO,
usage, introduction, etc.)
do test/example doc stuff for parsing and typechecking
parse transform pretty print example: basics of approach in chaos - do
some notes on how this can be improved
hssqlsystem commands reference
== tidy files, comment work
particularly ag files, redo comments, review all code for crufty bits
check todos at top of files: move to bottom
== nullability
== example code gen for typesafe access
== api review
tidyups, renames - rename everything pending, review public api
function names (particularly environment->catalog, and
localidentifierbindings to localenvironment)
get rid of inconsistent suffixes in hssqlsystem and command components
== error handling work
remove all calls to error
try to catch all places where errors are being ignored (mainly in
the type checking: add these errors to the aast)
== roundtrip tests -> report
== get chaos stuff working again
get chaos sql loading via hssqlppp again so can run roundtrip tests on
== module system, show used catalog
== better pretty printers for annotation types
== extensions code tidy?
== annotation generics style
== non sql syntax extensions work
== catalog ish: triggers, sequences, constraints, index, defaults
== haddock work
== do makefile type thing for development and deployment of db projects
maybe these:
builddocs work
hssqlsystem commands
better cat diff
show used catalog
finish intro docs
haskell intro docs
other docs
haddock in command components
testbattery
extensions - tidy code
annotation generics: tidy code
pretty printer for annotation
ppshow less newlines
better syntax for extensions
modules extension + public/private
syntax extension with non sql syntax using preprocessor?
multiline strings with string type e.g. sql
some sort of report mode to parse, typecheck and roundtrip some sql,
and report back what works and what doesn't
work on parse errors
long term todos
milestones for next release
todo for chaos (extensions, etc.)
divide todo categories
investigation, end user docs, internal docs, hygiene/maintenance,
testing, new functionality, bug fixes
docs: motivation/genesis, history?, high level criteria for 0.1
think about dependencies: make hdbc/pg dependency a flag?, split out
exes and tests somehow?
= documentation todo
write command processor thingy for builddocs
add extra commands to hssqlsystem for intro
finish intro
add haskell intro which uses the commandcomponents instead
other documentation files:
reference for hssqlsystem
hssqlsystem reference is generated from hssqlsystem.lhs:
inside this file is the reference information for each
command, plus example usage, so these appear next to the actual
source. then when building docs, filter this file first to leave
only the reference docs, then commandize and pandoc it
installation guide
do haddock for commandcomponents so it has examples and stuff
= New functionality
get checkbig mostly working: not trying to get no type errors or
differences, but to get all the different aspects of the tests
written and running so can see from the output what remains to be
fixed
main issues:
databaseloader isn't working on the chaos sql files anymore, I
think this is an issue with the pretty printer producing invalid
sql
finish extensions: for the tests to mostly work, need to flesh out
the current extension implementations which are mainly stubs
add missing items to catalog: triggers, sequences, constraints,
indexes, defaults
= Amending existing functionality
add dbms database name, username and password support
review holes:
round off parsing support for sql - make a list of what
needs to be added
should try to get as much typechecking working on what parses as
possible
work on organising the type checking tests, and make the tests much
more comprehensive
= Housecleaning
rename Environment type to Catalog
change type annotation from Type to either typecheckfailed Type so can
use monad style to propagate typecheckfaileds
work on error handling: try to add a lot better error reporting:
a few errors are being ignored rather than added to the aast, also
try to change all the calls to error into eithers to be added to
the aast
review commands in hssqlppp: mainly review names and arguments for
consistency (i.e. make sure every command accepts first a
databasename and then a list of sql files where this makes sense),
also redo annotate source so it can be run multiple times on the
same source, and run to update the annotation comments when the
source is changed. Maybe think about better error reporting?
tidy up code, refactor and add lots of comments, particularly in the
ag files
= Documentation
work on haddock documentation
find way to do nice html example documentation: short
intro/tour/tutorial to give people a flavour of what the
hssqlsystem exe can do
want to be able to show what the code can do easily and debug
annotations - so do a pretty printer which works the following way:
output source code on left side of a table (with syntax
highlighting, hyperlinks, etc? - use html?) and on the right output
the interesting annotations pretty printed, draw lines from the
pretty printed annotations to the sourcepositions in the source on
the left. Looks like might be able to do this using gtk:
http://www.bravegnu.org/gtktext/x498.html - maybe convert to png to
allow viewing in a web browser
= other issues to maybe fix:
check associativity of combine selects and join trefs without parens -
write tests using pg_dump output
ids in joins issue
redo order by parsing/ast to add direction to each element
write some notes on hacking: source overview rewrite, file overview
rewrites, and rough howto on adding new constructs: altering
parser, altering ast and altering type checking process
typecheck old and new pseudo records in trigger functions
explore creating template syntax for simple expansions of function
calls to make writing some extensions easier
makefile stuff?
plpgsql stuff: blocks, assignments in declarations, strict into,
labels,loops,continue
look into running test suite from cabal
todo for checkbig:
get initial catalog tests passing:
still some extra table triggers: think it may that foreign keys that
pg supports directly and primary keys don't add a table trigger
name/text result set resolution issue
uniquify constraint names support
trim constraint names support
distance function return type issue
plpgsql language functions
support multiple files in annotate source and check source
match pgdump output for adding explicit parens e.g. in select unions
and table refs, add tests for associativity in the absence of
parens
tidy up code and docs as usual - lots of work needed on type checking
code
redo catalog stuff (see below)
find some way to produce documentation: want motivation, examples,
etc. in nice html
get chaos pgdump output parsing and partially type checking, todo:
two parsing issues: not not, and count(distinct *)
type checking issues exposed:
old,new in trigger functions
old todoage:
================================================================================
== 2nd next release
fix catalog type, lots of work to make more robust:
rename environment to catalog
better pretty printer for catalog, and for bindings
add annotations to catalog, so can see where bits came from
do uniqueness testing e.g. disallow two tables with same name, two
attributes with same name, etc.
fix tests using new catalog approach: don't want to have tests which
use sql to alter catalog then use sql to test the catalog changes
have been made - use separate tests (more unit-y tests)
=== types and internal errors handling
move type check failed to left of either, make node types an either
when there is a type error, add that as separate annotation, and add
type as left typecheckfailed. This way can use the haskell type system
to check we are chaining type check failed properly.
look at all the places where type lists are wrapped in
unnamedcomposite and get rid of most of them - just use the type
list directly.
Look at all the contexts where pg types can be used, and split into
separate haskell types, and use sums to put them back together
to use the haskell type system better e.g. different set of types
can be used for an attribute in a table, for a function parameter,
function return type, variable declaration, etc. - want to use the
haskell type system to enforce this and get rid of a load of
runtime checks (mainly the unwrap setof, etc. in typetype).
make sure errors aren't ignored, lots of places where this currently
happens, e.g. updating environment - these errors need to end up in
the ast. Get all the env functions to return eithers, and stick
errors from these in the ast. Convert error calls to either return
type and stick these errors in the ast.
=== parse/typecheck errors to user
pretty printers for type check failed, work on error messages from parsing
investigate alternative parser libs which can e.g. continue parsing
after an error to provide more parse errors at once.
write a show for parsec errors which formats the lex tokens and
expected lists properly (was broken when moved to the separate
lexer)
================================================================================
== other soon todos
not sure which ones should be done before 0.1 release. rough guide is
0.1 release should be something that regular sql programmers can
try out/use without being overwhelmed with half finished/ missing
bits, and actually see something useful.
=== schemas
think of a way to handle schemas, what about changes to schema search
path mid code.
=== catalog and bindings annotation
want to add the catalog and current catalog to the annotations, so
e.g. we can pretty print what exact updates are made to the catalog
from a ddl statement
Add the current bindings as annotation everywhere, this can be used by
the ambiguous identifier checks, etc., and to understand errors
also: add annotations to the catalog so we can see where an entity in
the catalog has come from, what update and what source line
add similar to bindings, so each binding annotation contains a little
explanation for where each binding has come from (e.g. identifier a
with type b comes from this tref at source position x,y)
=== casting annotation
for each casting operation, add a log which explains what the relevant
algorithm has done to either find the appropriate cast or what it's
considered before failing. (e.g. can provide a list of steps taken
with the candidate lists in the findcallmatch routine, might be
useful for errors when it can't find a match, and if it is calling
an unexpected function instead of the one you want).
add the prototype of the function actually matched at each funcall
site
=== definition annotations
add sourcepos/reference to definition where each item is referenced
(table, function, view, identifier binding, etc.)
=== documentation produce
=== null checking
null treatment
Basic motivation is to keep nulls carefully walled off, controlled,
and be able to catch them when they sneak back into expressions,
etc.. For each value, etc. we determine statically if it might be
null. This can be done for return types of functions, fields in a
select expression, etc.. (will do mappings e.g. if a functions
inputs are all non null, then the output is non null, etc.). Once
this is working ok, the second stage is to implement the anti null
warnings/ errors.
Allow nulls in tables, outer joins, in coalesce, to be produced by
selects (maybe add or remove from this allowed list, maybe make it
configurable on a per project basis).
Never allow nulls to be an argument to a function call, (including
ops, keyword ops, etc.). So every time you have a field being used
in an expression and it cannot be statically verified to be non
null, you have to insert a coalesce or fix it in some other way.
So nulls can still be used to represent optional values, n/a,
etc.. and output to clients doing selects, but there is no need to
grapple with:
* 3vl (or whatever it is that sql uses instead),
* what the result of a function call is if the some or all the
arguments are null,
* what the result of a sum aggregate is if some of the values are null,
* etc.,
because none of these things are allowed.
=== do sql support todo
fill in the main gaps, some notes below
want to cover '90% of regularly used sql', no attempt for v0.1 to be
comprehensive or nearly comprehensive
=== shadowed identifier test
use the bindings annotation to add warnings
=== modules extension
stage 1: set the current module with a function call
do creating the module and adding objects to the current module using
extension instead of the sql functions used at the moment
find a way to annotate the source with modules stuff, want to use it
for documentation
stage 2: add import/export stuff and check this
=== other extensions for chaos
mark generated code with annotation: make annotation types open with type
class or something? - can get rid of typeerrora, statementinfoa
wrappers this way
check for all ddl, etc. run in functions/executes and turn into
extensions so the type checker can see these things
out of order definitions: want to write defs in any order and
have the extension reorder them into something which can be loaded
into postgres
finish create_var extension: add constraints
constraint system: make a bunch of stuff automatic, like using 'pg
accelerators'
add notify triggers automatically, including for views
convert the sql tests (maybe some of the haskell tests) into static
checks on the ast
find a better way to work with the tables with null fields:
initial values - the copy statements are completely unreadable and
uneditable
what else?
crufty bits to try and simplify: turn sequence progression, action
valid tables, ai
see if can make the multiple update hack more automatic
=== development/deployment support:
build file for development, just lists sql files in order, and lists
which extensions are used
deployment: use system to load all sql via extensions into pg, then
to a regular pg dump which can then be deployed.
=== crufty bits to tidy
ag files constantly need fixing up for documentation and code clarity
syb stuff is a bit messy in astannotations (maybe use uniplate here to
reduce dependencies?)
tidy the annotating functions in astinternals, implement then in terms
of eachother
review local bindings use and decide how much logic to implement
inside the local bindings module rather than eg. tablerefs.ag
want to stop having to write explicit backtrees everywhere. The key to
fixing this is some way to read the existing annotation and supply
an altered annotation within uuagc code
think about how doing multiple passes could help (e.g. star
expansion).
think about where can use quickcheck
review when function bodies are typechecked - see if can do better
annotatesource function is a mess, also, make it overwrite it's own
annotations instead of adding them so you can run it over a file
multiple times.
=== testing
work on making the tests more comprehensive, want a test for each
branch/option where feasible
=== write some automated tests for various roundtrips
add script to test chaos sql in various ways: parse roundtrip, load
into database via psql and via hssqlppp comparison, and check
catalog consistency
=== do way to produce aast from live database
use psql dump, and then can do type checks, lint checks, produce
documentation
=== api
review/work on api and do haddock
=== prepare code for use by non haskell programmers + user docs
provide installation instructions
do usage documentation file, want to run code embedded in doc source
and have results outputted to html output.
=== prepare code for use by haskell client programmers:
user docs as above for using the parser, the extension system, the
annotator and annotations, examples for generating sql and writing
extensions, etc.
=== ? place holder statements
=== precedence and operator parsing
add missing operators to parser
see if @ as prefix and infix just works in parsec
some a_expr and b_expr support properly by looking at pg grammar file
---------
some new random notes
literal sql mode, with workspace idea: can write code/sql inline
in the source, and run it against a database and view the results
inline like in some text books where code is executed then the results
printed inline into the result - so this is a partial replacement for
ghci, emacs haskell-mode ghci, psql, maybe bash, maybe ide, and
also a runtime diagnosis tool which can be embedded into production
exes. Pretty ambitious, need to focus on useful and easy features.
variant:
hssqlppp workspace: text view can type in commands and get results
back, create multiple workspaces and they're persistent, maybe
a wrapper round ghci but not conventional repl style, more like
entering elisp and evaluating it in emacs? want to use
as replacement for the ghci thing in emacs, and as a shell for working with
the ast stuff and actual databases
use template haskell to simplify generics in extensions: cases and
replacements are a bit verbose and difficult to write and read?
= parse and/or type check todo list:
(rough/incomplete)
"identifier"
6.5e-5
type 'string' style type cast
[:] slice
missing keyword ops
default template1 operators should all parse
agg(all expr) agg(distinct expr), agg(*)
window frame clauses, named windows
parse inside string literals when cast, for common types
multidimensional arrays
default values
serial
make sure can type check everything that parses
constraint names
provide list of keys in info for create/alter table: include unique
not null and serials
type check fks, and other constraints
alter table: add/remove column
constraint
default value
column type
rename column
rename table
what other alters/creates
views, functions, operators, types, domains, triggers, rules
selects:
implicit joins
group by, having + group by with unaggregated and aggregated fields
distinct, on
order by - do properly
limit, offset
with queries
type modifiers
data type names with spaces in them
timestamps
schemas
alternative text for true and false
enums
geometric types weird syntax
composites: selector variants, rowctors, component get/update
do all keyword and template1 operators
any/some/all subqueries and arrays
check over rowwise comparisons
indexes: create/alter/drop
another todo list:
catalog:
constraints, pg_constraint
triggers
rules
indexes
add drops for all creates
review which alters to add
default columns values + insert checking
support pks,fks better
support serial properly
type modifiers
sequences
enums
listen, notify, pg_listener
blocks in plpgsql, assignments in declarations, not null checking
strict in into
lables, loops, exit, continue
raise % counts
create schema
once all this is done, work on documenting precisely what is supported
and what isn't.
================================================================================
example for generating sql code from haskell using the ast
get database loader and typesafe access generators good enough to use
in chaos
example usage of each of these
look at the error message formatting, particularly try to fix the
parser errors so they make more sense
add annotation field to most ast nodes, store type and source
positioning in this field, fix parser to add lots of accurate
positioning information when parsing.
make sure the lint process works on text dumps of databases.
try checking the sample databases: http://pgfoundry.org/projects/dbsamples/
================================================================================
some syntax todo, not organised:
------------
add support for following sql syntax (+ type checking)
alter table, common variations
create index
create rule
create trigger
+ drops for all creates
+ maybe alters?
ctes
loop, exit, labels
easy ones: transactions, savepoints, listen
prepare, execute + using
some more:
create or replace
alter table
transactions: begin, checkpoint, commit, end, rollback
cursors: declare, open, fetch, move, close, where current of
copy - parse properly
create database
create index
create rule
create trigger + plpgsql support
grant,revoke
listen, notify, unlisten
prepare, execute
savepoint, release savepoint, rollback to savepoint
set, reset
set constraints
set role
set transaction
correlated subquery attrs
plpgsql
blocks which aren't at the top level of a function
% types
strict on intos
not null for var defs
exception
execute using
get diagnostics
return query execute
raise missing bits
out params
elsif
loop
exit
labels
reverse, by in for
for in execute
expressions:
process string escapes, support dollar quoting and other quoting more
robustly in the pretty printer
full user operator support (?)
fix expression parser properly to handle things like between - see
grammar in pg source for info on how to do this
[:] array slices
aggregate: all and distinct
multi dimensional arrays: selectors and subscripting
missing keyword operators
datetime extract
time zone
subquery operators: any, some, all
in general, parsing operators is wrong, the lexer needs to be able to
lex sequences of symbols into single/multiple operators correctly,
what happens at the moment is a kludge, also, general operator
parsing will change how operators are represented in the ast
================================================================================
some other random ideas:
more tests:
add parse and type check roundtrips automated tests for chaos sql
add parse check which parses a single statement, then loads it into
pg, pg_dumps it, then parses the output to see if it is the same:
since pg sticks plenty of parens in, this should test the
associativity and precedence. pg also rewrites some of the sql so
will have to match that
parser, converter and pretty printer for explain output, want to view
how a query is executed in human readable pseudocode. Add lint type
checks, etc. to this, which can suggest ways to rewrite the query
to get better performance. Another idea is to make the dependencies
on the values in the tables more explicit, so you can see how much
the data can change before another plan is chosen, or you can see a
bad assumption about the kind of data the query will be run on.
incorporate pg regression test sql into parsing and type checking
tests - can run the roundtrip tests on them, check the
documentation generator, etc.
redo cabal file to add compile time options: exes, pg support, tests
or split into separate packages?
work on error reporting, add tests for malformed sql, already some
tests and plan for more for sql which fails type checks.
want to report multiple parse errors, perhaps can bodge this because
of the property that ';' can only appear inside a string or
comment, or otherwise at the end of a statement, so add some code
to jump to the next end of statement looking ';' and continue to
parse to end of file in an attempt to catch at least some further
syntax errors
get property checker working again - one problem is that the pretty
printer will reject some asts (which the parser cannot
produce), and the parser will probably reject some invalid sql that
the pretty printer will happily produce from some asts.
plpgsql on 'roids:
write libraries in haskell, and then write syntax extensions for
plpgsql using the extension mechanism to access these libs from
extended plpgsql e.g. ui lib written in haskell, accessed by syntax
extensions in plpgsql then can write the database and ui all in the
same source code in the same language, with first class support for
properly typed relation valued expressions, avoiding multiple
languages and mapping/'impedance mismatch' between database types
and types in the language you write the ui in.