jsontsv (empty) → 0.1.0.0
raw patch · 8 files changed
+384/−0 lines, 8 filesdep +aesondep +attoparsecdep +basesetup-changed
Dependencies added: aeson, attoparsec, base, bytestring, containers, scientific, text, unordered-containers, vector
Files
- .gitignore +1/−0
- LICENSE +24/−0
- Main.hs +128/−0
- README.md +92/−0
- Setup.hs +2/−0
- example.json +22/−0
- jsontsv.cabal +30/−0
- notes +85/−0
+ .gitignore view
@@ -0,0 +1,1 @@+out.json
+ LICENSE view
@@ -0,0 +1,24 @@+The MIT License (MIT)++Copyright (c) 2014 Daniel Choi <dhchoi@gmail.com>++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in+all copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN+THE SOFTWARE.+++
+ Main.hs view
@@ -0,0 +1,128 @@+{-# LANGUAGE OverloadedStrings #-}+module Main where+import Data.Aeson+import Data.Monoid+import qualified Data.Map.Strict as M+import Data.Text (Text)+import qualified Data.Text.Encoding as T (decodeUtf8)+import Data.List (intersperse)+import qualified Data.Text as T+import qualified Data.Text.Lazy.IO as TL+import Data.Maybe (catMaybes)+import Control.Applicative+import Data.ByteString.Lazy as BL hiding (map, intersperse)+import qualified Data.ByteString.Lazy.Char8 as BS+import Data.Attoparsec.Lazy as Atto hiding (Result)+import Data.Attoparsec.ByteString.Char8 (endOfLine, sepBy)+import qualified Data.Attoparsec.Text as AT+import qualified Data.HashMap.Lazy as HM+import qualified Data.Vector as V+import Data.Scientific +import System.Environment (getArgs)+import qualified Data.Text.Lazy.Builder as B+import qualified Data.Text.Lazy.Builder.Int as B+import qualified Data.Text.Lazy.Builder.RealFloat as B++main = do+ x <- BL.getContents + let xs :: [Value]+ xs = decodeStream x+ (ks:_) <- getArgs + let ks' = parseKeyPath $ T.pack ks+ -- Prelude.putStrLn $ "key Paths " ++ show ks'+ mapM_ (TL.putStrLn . B.toLazyText . evalToLineBuilder ks') xs++decodeStream :: (FromJSON a) => BL.ByteString -> [a]+decodeStream bs = case decodeWith json bs of+ (Just x, xs) | xs == mempty -> [x]+ (Just x, xs) -> x:(decodeStream xs)+ (Nothing, _) -> []++decodeWith :: (FromJSON a) => Parser Value -> BL.ByteString -> (Maybe a, BL.ByteString)+decodeWith p s =+ case Atto.parse p s of+ Atto.Done r v -> f v r+ Atto.Fail _ _ _ -> (Nothing, mempty)+ where f v' r = (\x -> case x of + Success a -> (Just a, r)+ _ -> (Nothing, r)) $ fromJSON v'++parseKeyPath :: Text -> [KeyPath]+parseKeyPath s = case AT.parseOnly pKeyPaths s of+ Left err -> error $ "Parse error " ++ err + Right res -> res++spaces = many1 AT.space++pKeyPaths :: AT.Parser [KeyPath]+pKeyPaths = pKeyPath `AT.sepBy` spaces++pKeyPath :: AT.Parser KeyPath+pKeyPath = AT.sepBy1 pKeyOrIndex (AT.takeWhile1 $ AT.inClass ".[")++pKeyOrIndex = pIndex <|> pKey++pKey = Key <$> AT.takeWhile1 (AT.notInClass " .[")++pIndex = Index <$> AT.decimal <* AT.char ']'++type KeyPath = [Key]+data Key = Key Text | Index Int deriving (Eq, Show)++evalToLineBuilder :: [KeyPath] -> Value -> B.Builder +evalToLineBuilder ks v = mconcat $ intersperse (B.singleton '\t') $ map (flip evalToBuilder v) ks++evalToList :: [KeyPath] -> Value -> [Text]+evalToList ks v = map (flip evalToText v) ks++evalToBuilder :: KeyPath -> Value -> B.Builder+evalToBuilder k v = valToBuilder $ evalKeyPath k v++evalToText :: KeyPath -> Value -> Text+evalToText k v = valToText $ evalKeyPath k v++-- evaluates the a JS key path against a Value context to a leaf Value+evalKeyPath :: KeyPath -> Value -> Value+evalKeyPath [] x@(String _) = x+evalKeyPath [] x@Null = x+evalKeyPath [] x@(Number _) = x+evalKeyPath [] x@(Bool _) = x+evalKeyPath [] x@(Object _) = x+evalKeyPath [] x@(Array v) = + let vs = V.toList v+ xs = intersperse "," $ map (evalToText []) vs+ in String . mconcat $ xs+evalKeyPath (Key key:ks) (Object s) = + case (HM.lookup key s) of+ Just x -> evalKeyPath ks x+ Nothing -> Null+evalKeyPath (Index idx:ks) (Array v) = + let e = (V.!?) v idx+ in case e of + Just e' -> evalKeyPath ks e'+ Nothing -> Null+evalKeyPath ((Index _):_) _ = Null+evalKeyPath _ _ = Null++valToBuilder :: Value -> B.Builder+valToBuilder (String x) = B.fromText x+valToBuilder Null = B.fromText "null"+valToBuilder (Bool True) = B.fromText "t"+valToBuilder (Bool False) = B.fromText "f"+valToBuilder (Number x) = + case floatingOrInteger x of+ Left float -> B.realFloat float+ Right int -> B.decimal int+valToBuilder (Object _) = B.fromText "[Object]"++valToText :: Value -> Text+valToText (String x) = x+valToText Null = "NULL"+valToText (Bool True) = "T"+valToText (Bool False) = "F"+valToText (Number x) = + case floatingOrInteger x of+ Left float -> T.pack . show $ float+ Right int -> T.pack . show $ int+valToText (Object _) = "[Object]"+
+ README.md view
@@ -0,0 +1,92 @@+# jsontsv++A simple tool to transform JSON into tab-separated line-oriented output+amenable to downstream Unix text processing. ++## Synopsis++input.json:++```json+{+ "title": "Terminator 2: Judgement Day",+ "year": 1991,+ "stars": [+ "Arnold Schwarzenegger",+ "Linda Hamilton"+ ],+ "ratings": {+ "imdb": 8.5+ }+}+{+ "title": "Interstellar",+ "year": 2014,+ "stars": [+ "Matthew McConaughey",+ "Anne Hathaway"+ ],+ "ratings": {+ "imdb": 8.9+ }+}+```++ jsontsv 'title year stars ratings.imdb' < input.json++Outputs this tab-separated text:++```tsv+Terminator 2: Judgement Day 1991 Arnold Schwarzenegger,Linda Hamilton 8.5+Interstellar 2014 Matthew McConaughey,Anne Hathaway 8.9+```++## Setup++From the project directory, ++ cabal install++Make sure the installed executable is on your PATH.++## Usage++Input should be a stream of JSON objects with mostly uniform keys, separated by+whitespace such as newlines. If the objects are wrapped in a JSON array at the+top level, use the `jq` tool by Stephan Dolan to unwrap the objects, e.g.: ++ curl -s "https://api.github.com/users/danchoi/repos?type=owner&sort=created&direction=desc" \+ | jq '.[]' | jsontsv 'id name stargazers_count open_issues_count' ++outputs++ 27397673 jsontsv 0 0+ 26033118 ngrender 24 1+ 25832026 rdoc 0 0+ 24756523 treehtml 0 0+ 24022588 heistexamples 0 0+ 24022042 hxtexamples 0 0+ 24005242 jdiff 0 0+ 23997156 https-types 0 0+ 22763562 podcasting 0 0+ 19294791 vimscript 3 0++JSON leaf values are printed as follows: ++* Strings and numbers were copied to output.+* Boolean values are output as `t` or `f`.+* null is printed as `null`+* Arrays of leaf values are concatenated into a single comma-separated string++## Nested keys++ jsontsv 'title duplicates.Rental.HD duplicates.Rental.SD' < input.json++## Using a file to designate columns:++ jsontsv -f keys < input.json++## Concatenating fields, truncating fields, etc.++This should be done downstream using a tool like AWK.+
+ Setup.hs view
@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain
+ example.json view
@@ -0,0 +1,22 @@+{+ "title": "Terminator 2: Judgement Day",+ "year": 1991,+ "stars": [+ "Arnold Schwarzenegger",+ "Linda Hamilton"+ ],+ "ratings": {+ "imdb": 8.5+ }+}+{+ "title": "Interstellar",+ "year": 2014,+ "stars": [+ "Matthew McConaughey",+ "Anne Hathaway"+ ],+ "ratings": {+ "imdb": 8.9+ }+}
+ jsontsv.cabal view
@@ -0,0 +1,30 @@+-- Initial jsontsv.cabal generated by cabal init. For further documentation,+-- see http://haskell.org/cabal/users-guide/++name: jsontsv+version: 0.1.0.0+synopsis: JSON to TSV transformer+homepage: https://github.com/danchoi/jsontsv+description: Transforms JSON into tab-separated line-oriented output, for easier processing in Unix-style pipelines.+license: MIT+license-file: LICENSE+author: Daniel Choi+maintainer: dhchoi@gmail.com+copyright: (c) 2014 Daniel Choi+category: Text+build-type: Simple+cabal-version: >=1.10++executable jsontsv+ main-is: Main.hs+ build-depends: base >=4.7 && <4.8+ , aeson >= 0.8.0.0+ , bytestring+ , attoparsec >= 0.12.1.0+ , containers+ , unordered-containers+ , text >= 1.2.0.0+ , vector+ , scientific+ default-language: Haskell2010+ ghc-options: -O2
+ notes view
@@ -0,0 +1,85 @@+http://hackage.haskell.org/package/aeson-0.8.0.1++decode = decodeWith jsonEOF fromJSON++decode :: (FromJSON a) => L.ByteString -> Maybe a+decode = decodeWith jsonEOF fromJSON++http://hackage.haskell.org/package/aeson-0.8.0.1/docs/src/Data-Aeson.html#decode++import Data.Aeson.Parser.Internal (decodeWith, decodeStrictWith,+ eitherDecodeWith, eitherDecodeStrictWith,+ jsonEOF, json, jsonEOF', json')+++import qualified Data.Attoparsec.Lazy as L++http://hackage.haskell.org/package/aeson-0.6.0.0/docs/src/Data-Aeson-Parser-Internal.html+++decodeWith :: Parser Value -> (Value -> Result a) -> L.ByteString -> Maybe a+decodeWith p to s =+ case L.parse p s of+ L.Done _ v -> case to v of+ Success a -> Just a+ _ -> Nothing++import qualified Data.Attoparsec.Lazy as L++https://hackage.haskell.org/package/attoparsec-0.12.1.2++import Data.Attoparsec.ByteString.Lazy++https://hackage.haskell.org/package/attoparsec-0.12.1.2/docs/Data-Attoparsec-ByteString-Lazy.html++jsonEOF :: Parser Value+jsonEOF = json <* skipSpace <* endOfInput+++++sepBy1 :: Alternative f => f a -> f s -> f [a]+sepBy1 p s = scan+ where scan = liftA2 (:) p ((s *> scan) <|> pure [])++scan :: s -> (s -> Char -> Maybe s) -> Parser Text++A stateful scanner. The predicate consumes and transforms a state argument, and each transformed state is passed to successive invocations of the predicate on each character of the input until one returns Nothing or the input ends.++This parser does not fail. It will return an empty string if the predicate returns Nothing on the first character of input.++Note: Because this parser does not fail, do not use it with combinators such as many, because such parsers loop until a failure occurs. Careless use will thus result in an infinite loop.++Control.Applicative liftA2 :: Applicative f => (a -> b -> c) -> f a -> f b -> f c+++http://hackage.haskell.org/package/attoparsec-0.12.1.2/docs/Data-Attoparsec-Text.html#v:Partial++Incremental input++attoparsec supports incremental input, meaning that you can feed it a Text that+represents only part of the expected total amount of data to parse. If your+parser reaches the end of a fragment of input and could consume more input, it+will suspend parsing and return a Partial continuation.++Supplying the Partial continuation with another string will resume parsing at+the point where it was suspended, with the string you supplied used as new+input at the end of the existing input. You must be prepared for the result of+the resumed parse to be another Partial continuation.++To indicate that you have no more input, supply the Partial continuation with+an empty Text.++Remember that some parsing combinators will not return a result until they+reach the end of input. They may thus cause Partial results to be returned.++If you do not need support for incremental input, consider using the parseOnly+function to run your parser. It will never prompt for more input.++Note: incremental input does not imply that attoparsec will release portions of+its internal state for garbage collection as it proceeds. Its internal+representation is equivalent to a single Text: if you feed incremental input to+an a parser, it will require memory proportional to the amount of input you+supply. (This is necessary to support arbitrary backtracking.)++