diff --git a/CHANGES.txt b/CHANGES.txt --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,5 +1,8 @@ Changelog for Hoogle (* = API change, @ = database format change) +5.0.18.2, released 2021-09-01 + Compatibility with extra-1.7.10 + #370, add --jsonl to produce JSON on separate lines 5.0.18.1, released 2020-12-27 #365, add flag to disable network tests 5.0.18, released 2020-07-19 diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright Neil Mitchell 2004-2020. +Copyright Neil Mitchell 2004-2021. All rights reserved. Redistribution and use in source and binary forms, with or without diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,38 +1,21 @@ -# Hoogle [![Hackage version](https://img.shields.io/hackage/v/hoogle.svg?label=Hackage)](https://hackage.haskell.org/package/hoogle) [![Stackage version](https://www.stackage.org/package/hoogle/badge/nightly?label=Stackage)](https://www.stackage.org/package/hoogle) [![Build status](https://img.shields.io/github/workflow/status/ndmitchell/hoogle/ci.svg)](https://github.com/ndmitchell/hoogle/actions) - -## Hoogle 4 vs Hoogle 5 - -Hoogle is in the middle of a transition from version 4 to version 5. - -Hoogle 4 is at https://www.haskell.org/hoogle/. It searches the [Haskell platform](https://www.haskell.org/platform/contents.html) as-of 2013. It has good type search. - -Hoogle 5 is at https://hoogle.haskell.org/ and on Hackage, so will be obtained by `cabal install hoogle`. It searches [Stackage](https://www.stackage.org/) and is updated daily. It has weaker type search, with various tickets to improve that. - -## Other stuff (somewhat outdated) - - ----------- - -**This page describes how Hoogle 5 might work, and has not yet been fully implemented.** - ----------- +# Hoogle [![Hackage version](https://img.shields.io/hackage/v/hoogle.svg?label=Hackage)](https://hackage.haskell.org/package/hoogle) [![Stackage version](https://www.stackage.org/package/hoogle/badge/nightly?label=Stackage)](https://www.stackage.org/package/hoogle) [![Build status](https://img.shields.io/github/workflow/status/ndmitchell/hoogle/ci/master.svg)](https://github.com/ndmitchell/hoogle/actions) -Hoogle is a Haskell API search engine, which allows you to search many standard Haskell libraries by either function name, or by approximate type signature. To experiment, visit the online version at http://haskell.org/hoogle. +Hoogle is a Haskell API search engine, which allows you to search many standard Haskell libraries by either function name, or by approximate type signature. The online version can be found at https://hoogle.haskell.org/ and searches [Stackage](https://www.stackage.org/). -* **Online version:** https://www.haskell.org/hoogle/ +* **Online version:** https://hoogle.haskell.org/ * **Hackage page:** https://hackage.haskell.org/package/hoogle -* **Source code:** http://github.com/ndmitchell/hoogle +* **Source code:** https://github.com/ndmitchell/hoogle * **Bug tracker:** https://github.com/ndmitchell/hoogle/issues ## Hoogle Use Hoogle can be used in several ways: -* **Online**, with the web interface at http://haskell.org/hoogle -* **In [IRC](http://haskell.org/haskellwiki/Haskell_IRC_channel)**, using the [Lambdabot](http://haskell.org/haskellwiki/Lambdabot) plugin with `@hoogle` and `@hoogle+` +* **Online**, with the web interface at https://hoogle.haskell.org/ +* **In [IRC](https://wiki.haskell.org/IRC_channel)**, using the [Lambdabot](https://wiki.haskell.org/Lambdabot) plugin with `@hoogle` and `@hoogle+` * **From `emacs`**, by means of [`engine-mode`](https://github.com/hrs/engine-mode) -* **[Installed locally](https://github.com/ndmitchell/hoogle/blob/master/docs/Install.md)**, with either a command line or in a browser -* **[As a developer](https://github.com/ndmitchell/hoogle/blob/master/docs/API.md)**, through Haskell or JSON APIs. +* **[Installed locally](./docs/Install.md)**, with either a command line or in a browser +* **[As a developer](./docs/API.md)**, through Haskell or JSON APIs. # Searches @@ -50,7 +33,7 @@ ## Scope -By default, searches look at the [Haskell Platform](http://hackage.haskell.org/platform) and [Haskell keywords](http://haskell.org/haskellwiki/Keywords). However, all [Hackage](http://hackage.haskell.org) packages are available to search. As some examples: +By default, searches look at the [Haskell Platform](https://www.haskell.org/platform/) and [Haskell keywords](https://wiki.haskell.org/Keywords). However, all [Stackage](https://stackage.org) packages are available to search. As some examples: * `mode +cmdargs` searches only the "cmdargs" package * `file -base` searches the Haskell Platform, excluding the "base" package @@ -69,14 +52,14 @@ To invoke Hoogle type: - hoogle "[a] -> [b]" + $ hoogle "[a] -> [b]" Note the quotes, otherwise you will redirect the output to the file [b]. To ensure you have data files for the Hackage modules, you will first need to type: - hoogle generate + $ hoogle generate Which will download and build Hoogle databases. @@ -86,21 +69,13 @@ ## Chrome Integration -**As a keyword search:** With a keyword search you can type `h map` directly into the location bar to perform a Hoogle search. Go to the [Hoogle website](http://haskell.org/hoogle/) in Chrome, right-click in the Hoogle search field and select "Add as a search engine...". Give it a keyword such as "h". - +**As a keyword search:** With a keyword search you can type `h map` directly into the location bar to perform a Hoogle search. Go to the [Hoogle website](https://hoogle.haskell.org/) in Chrome, right-click in the Hoogle search field and select "Add as a search engine...". Give it a keyword such as "h". ## Firefox Integration -**From the search bar:** Go to the [Hoogle website](http://haskell.org/hoogle/) in Firefox and click on the drop-down arrow at the left of the search bar, and select the "Add Hoogle" option. Click the arrow again to select Hoogle as your search engine. - -**As a keyword search:** With a keyword search you can type `h map` directly into the location bar to perform a Hoogle search. Go to the [Hoogle website](http://haskell.org/hoogle/) in Firefox, right-click in the Hoogle search field and select "Add a Keyword for this Search...". Given it a keyword such as "h". - -If you want to search for special symbols in Firefox keyword search, modify the keyword search URL to be: `javascript:window.location.href="http://haskell.org/hoogle?q=" + encodeURIComponent("%s")` - - -## Firefox Ubiquity Integration +**From the search bar:** Go to the [Hoogle website](https://hoogle.haskell.org/) in Firefox and click on the `⋯` symbol at the right of the URL bar, and select the "Add Search Engine" option. Click the hoogle logo at the bottom of the completion dropdown when searching to perform a Hoogle search. -[Ubiquity](https://wiki.mozilla.org/Labs/Ubiquity) provides a graphical command-line for Firefox. To install the Ubiquity Hoogle command, visit the [this page](http://www.randomhacks.net/git/ubiquity/hoogle/) and click "Subscribe..." when asked whether you want to install it. Further information is available [here](http://www.randomhacks.net/articles/2008/09/01/ubiquitous-hoogle). +**As a keyword search:** With a keyword search you can type `h map` directly into the location bar to perform a Hoogle search. Go to the [Hoogle website](https://hoogle.haskell.org/) in Firefox, right-click in the Hoogle search field and select "Add a Keyword for this Search...". Given it a keyword such as "h". ## Others @@ -108,11 +83,9 @@ ### The Source Code -$ darcs get http://code.haskell.org/hoogle/ - -Contributions are most welcome. Hoogle is written in Haskell 98 + Heirarchical Modules, I do not wish to change this. Other than that, I'm pretty flexible about most aspects of Hoogle. The [http://code.google.com/p/ndmitchell/issues/list bug tracker] has many outstanding tasks, but please contact me if you have thoughts on doing something major to Hoogle, so I can give some advice. - + $ git clone https://github.com/ndmitchell/hoogle.git +Contributions are most welcome. Hoogle is written in Haskell 98 + Heirarchical Modules, I do not wish to change this. Other than that, I'm pretty flexible about most aspects of Hoogle. The [issue tracker](https://github.com/ndmitchell/hoogle/issues) has many outstanding tasks, but please contact me if you have thoughts on doing something major to Hoogle, so I can give some advice. # Background @@ -122,49 +95,49 @@ A lot of related work was done by Rittri [1] and Runciman [2] in the late 80's. Since then Di Cosmo [3] has produced a book on type isomorphisms. Unfortunately the implementations that accompanied the earlier works were for functional languages that have since become less popular. -1. [Mikael Rittri, Using Types as Search Keys in Function Libraries](http://portal.acm.org/citation.cfm?id=99384). Proceedings of the fourth international conference on Functional Programming languages and Computer Architecture: 174-183, June 1989. -2. [Colin Runciman and Ian Toyn, Retrieving reusable software components by polymorphic type](http://portal.acm.org/citation.cfm?id=99383). Journal of Functional Programming 1 (2): 191-211, April 1991. -3. [Roberto Di Cosmo, Isomorphisms of types: from lambda-calculus to information retrieval and language design](http://www.pps.jussieu.fr/~dicosmo/Publications/ISObook.html). Birkhauser, 1995. ISBN-0-8176-3763-X +1. [Mikael Rittri, Using Types as Search Keys in Function Libraries](https://doi.org/10.1145/99370.99384). Proceedings of the fourth international conference on Functional Programming languages and Computer Architecture: 174-183, June 1989. +2. [Colin Runciman and Ian Toyn, Retrieving reusable software components by polymorphic type](https://doi.org/10.1145/99370.99383). Journal of Functional Programming 1 (2): 191-211, April 1991. +3. [Roberto Di Cosmo, Isomorphisms of types: from lambda-calculus to information retrieval and language design](https://doi.org/10.1145/270563.571468). Birkhauser, 1995. ISBN-0-8176-3763-X -I have given several presentations on type searching all available from [my home page](http://community.haskell.org/~ndm/hoogle). +I have given several presentations on type searching all available from [my home page](https://ndmitchell.com/). -## Folders +## Project Structure -The folders in the distribution, and their meaning are: +The folders in the repository, and their meaning are: -data - tools to generate a hoogle data file +cbits - C implementation of the text search used by hoogle -docs - documentation on hoogle +docs - documention on hoogle -misc - presentations, icons, emacs scripts, logos +html - resources for hoogle's web front-end (html, css, javascript, images, etc.) -src - source code +misc - scripts, logos, sample data, etc. -web - additional resources for the web front end (css, jpg etc.) +src - haskell source code ## Similar Tools I was unaware of any similar tools before starting development, and no other tool has really influenced this tool (except the first on this list). Some related tools are: -* [Google](http://www.google.com/), the leader in online search -* [Hayoo](http://holumbus.fh-wedel.de/hayoo/hayoo.html), similar to Hoogle, but with less focus on type search -* [Krugle](http://www.krugle.com/), search code, but no Haskell :( -* [Cloogle](https://cloogle.org), for the [Clean](http://clean.cs.ru.nl) language +* [Google](https://www.google.com/), the leader in online search +* [Hayoo](https://hackage.haskell.org/package/Hayoo), similar to Hoogle, but with less focus on type search +* [Krugle](https://www.krugle.com/), search code, but no Haskell :( +* [Cloogle](https://cloogle.org), for the [Clean](https://clean.cs.ru.nl/Clean) language ## Acknowledgements -All code is all © [Neil Mitchell](http://community.haskell.org/~ndm/), 2004-present. The initial version was done over my summer holiday, and further work was done during my PhD. During Summer 2008 I was funded to full-time on Hoogle by [Google Summer of Code](http://code.google.com/soc/) with the [haskell.org](http://haskell.org/) mentoring organisation. Since then I have been working on Hoogle in my spare time. Various people have given lots of useful ideas, including my PhD supervisor [Colin Runciman](http://www.cs.york.ac.uk/~colin/), and various members of the [Plasma group](http://www.cs.york.ac.uk/plasma/). In addition, the following people have also contributed code or significant debugging work: +All code is all © [Neil Mitchell](https://ndmitchell.com/), 2004-present. The initial version was done over my summer holiday, and further work was done during my PhD. During Summer 2008 I was funded to full-time on Hoogle by [Google Summer of Code](https://summerofcode.withgoogle.com/) with the [haskell.org](https://www.haskell.org/) mentoring organisation. Since then I have been working on Hoogle in my spare time. Various people have given lots of useful ideas, including my PhD supervisor [Colin Runciman](https://www-users.cs.york.ac.uk/~colin/), and various members of the [Plasma group](https://www.cs.york.ac.uk/plasma/wiki/). In addition, the following people have also contributed code or significant debugging work: -* [Thomas "Bob" Davie](http://www.cs.kent.ac.uk/people/rpg/tatd2/) -* [Don Stewart](http://www.cse.unsw.edu.au/~dons/) +* Thomas "Bob" Davie +* Don Stewart * Thomas Jager -* [Gaal Yahas](http://gaal.livejournal.com/) -* [Mike Dodds](http://www-users.cs.york.ac.uk/~miked/) -* [Niklas Broberg](http://www.cs.chalmers.se/~d00nibro/) +* [Gaal Yahas](https://gaal.livejournal.com/) +* Mike Dodds +* Niklas Broberg * Esa Ilari Vuokko * Udo Stenzel -* [Henk-Jan van Tuyl](http://members.chello.nl/hjgtuyl/) +* [Henk-Jan van Tuyl](https://github.com/HJvT) * Gwern Branwen * Tillmann Rendel * David Waern diff --git a/hoogle.cabal b/hoogle.cabal --- a/hoogle.cabal +++ b/hoogle.cabal @@ -1,13 +1,13 @@ cabal-version: >= 1.18 build-type: Simple name: hoogle -version: 5.0.18.1 +version: 5.0.18.2 license: BSD3 license-file: LICENSE category: Development author: Neil Mitchell maintainer: Neil Mitchell -copyright: Neil Mitchell 2004-2020 +copyright: Neil Mitchell 2004-2021 synopsis: Haskell API Search description: Hoogle is a Haskell API search engine, which allows you to @@ -15,7 +15,7 @@ or by approximate type signature. homepage: https://hoogle.haskell.org/ bug-reports: https://github.com/ndmitchell/hoogle/issues -tested-with: GHC==8.10, GHC==8.8, GHC==8.6, GHC==8.4, GHC==8.2 +tested-with: GHC==9.0, GHC==8.10, GHC==8.8, GHC==8.6, GHC==8.4, GHC==8.2 extra-doc-files: README.md CHANGES.txt diff --git a/html/index.html b/html/index.html --- a/html/index.html +++ b/html/index.html @@ -33,6 +33,6 @@
#{body}
- + diff --git a/src/Action/CmdLine.hs b/src/Action/CmdLine.hs --- a/src/Action/CmdLine.hs +++ b/src/Action/CmdLine.hs @@ -8,14 +8,14 @@ whenLoud, whenNormal ) where +import Data.List.Extra +import Data.Version +import General.Util +import Paths_hoogle (version) import System.Console.CmdArgs import System.Directory import System.Environment import System.FilePath -import Data.List.Extra -import Data.Version -import General.Util -import Paths_hoogle(version) data Language = Haskell | Frege deriving (Data,Typeable,Show,Eq,Enum,Bounded) @@ -23,6 +23,7 @@ = Search {color :: Maybe Bool ,json :: Bool + ,jsonl :: Bool ,link :: Bool ,numbers :: Bool ,info :: Bool @@ -111,6 +112,7 @@ search_ = Search {color = def &= name "colour" &= help "Use colored output (requires ANSI terminal)" ,json = def &= name "json" &= help "Get result as JSON" + ,jsonl = def &= name "jsonl" &= help "Get result as JSONL (JSON Lines)" ,link = def &= help "Give URL's for each result" ,numbers = def &= help "Give counter for each result" ,info = def &= help "Give extended information about the first result" diff --git a/src/Action/Search.hs b/src/Action/Search.hs --- a/src/Action/Search.hs +++ b/src/Action/Search.hs @@ -1,4 +1,5 @@ -{-# LANGUAGE LambdaCase, RecordWildCards, ScopedTypeVariables, TupleSections #-} +{-# LANGUAGE LambdaCase, MultiWayIf, RecordWildCards, ScopedTypeVariables, + TupleSections #-} module Action.Search (actionSearch, withSearch, search @@ -8,17 +9,17 @@ ) where import Control.DeepSeq -import Control.Monad.Extra import Control.Exception.Extra +import Control.Monad.Extra import qualified Data.Aeson as JSON +import qualified Data.ByteString.Lazy.Char8 as LBS import Data.Functor.Identity import Data.List.Extra -import Text.Blaze.Renderer.Utf8 -import qualified Data.ByteString.Lazy.Char8 as LBS import qualified Data.Map as Map import Data.Maybe import qualified Data.Set as Set import System.Directory +import Text.Blaze.Renderer.Utf8 import Action.CmdLine import General.Store @@ -49,7 +50,9 @@ putStr $ targetInfo $ head res else do let toShow = if numbers && not info then addCounter shown else shown - if json then LBS.putStrLn $ JSON.encode $ maybe id take count $ map unHTMLtargetItem res else putStr $ unlines toShow + if | json -> LBS.putStrLn $ JSON.encode $ maybe id take count $ map unHTMLtargetItem res + | jsonl -> mapM_ (LBS.putStrLn . JSON.encode) $ maybe id take count $ map unHTMLtargetItem res + | otherwise -> putStr $ unlines toShow when (hidden /= [] && not json) $ do whenNormal $ putStrLn $ "-- plus more results not shown, pass --count=" ++ show (count'+10) ++ " to see more" else do diff --git a/src/Action/Server.hs b/src/Action/Server.hs --- a/src/Action/Server.hs +++ b/src/Action/Server.hs @@ -163,7 +163,7 @@ html = templateMarkup text = templateMarkup . H.string - tagOptions sel = mconcat [H.option !? (x `elem` sel, H.selected "selected") $ H.string x | x <- completionTags store] + tagOptions sel = mconcat [H.option Text.Blaze.!? (x `elem` sel, H.selected "selected") $ H.string x | x <- completionTags store] params = [("cdn", text cdn) ,("home", text home) diff --git a/src/Action/Test.hs b/src/Action/Test.hs --- a/src/Action/Test.hs +++ b/src/Action/Test.hs @@ -8,6 +8,7 @@ import Action.Server import Action.Generate import General.Util +import General.Web import Input.Item import Input.Haddock import System.IO.Extra @@ -22,6 +23,7 @@ actionTest Test{..} = withBuffering stdout NoBuffering $ withTempFile $ \sample -> do putStrLn "Code tests" general_util_test + general_web_test input_haddock_test query_test action_server_test_ diff --git a/src/General/Web.hs b/src/General/Web.hs --- a/src/General/Web.hs +++ b/src/General/Web.hs @@ -2,7 +2,7 @@ module General.Web( Input(..), - Output(..), readInput, server + Output(..), readInput, server, general_web_test ) where import Network.Wai.Handler.Warp hiding (Port, Handle) @@ -28,6 +28,7 @@ import Control.Exception.Extra import System.Time.Extra import General.Log +import General.Util import Prelude import qualified Data.ByteString.UTF8 as UTF8 @@ -35,7 +36,7 @@ data Input = Input {inputURL :: [String] ,inputArgs :: [(String, String)] - } deriving Show + } deriving (Eq, Show) readInput :: String -> Maybe Input readInput (breakOn "?" -> (a,b)) = @@ -45,7 +46,14 @@ parsePath = map Text.unpack . decodePathSegments . BS.pack - badPath = any (all (== '.')) . filter (/= "") + -- Note that there is a difference between URL paths + -- which split on / and only that and file paths where + -- an escaped %2f is equivalent to /. decodePathSegments + -- (correctly) only considers the former so here + -- we add an extra check that the result (which has unescaped %2f to /) + -- does not contain path separators. + badPath = any badSegment . filter (/= "") + badSegment seg = all (== '.') seg || any isPathSeparator seg args = parseArgs b parseArgs = map (UTF8.toString *** maybe "" UTF8.toString) . parseQuery @@ -178,3 +186,22 @@ OutputJavascript{} -> responseLBS status200 (("content-type","text/javascript") : secH) bs contentType = [(".html","text/html"),(".css","text/css"),(".js","text/javascript")] + +general_web_test :: IO () +general_web_test = do + testing "General.Web.readInput" $ do + let a === b = if a == b then putChar '.' else errorIO $ show (a,b) + readInput "abc" === Just (Input ["abc"] []) + readInput "/abc" === Just (Input ["abc"] []) + readInput "/abc/" === Just (Input ["abc", ""] []) + readInput "abc?ab=cd&ef=gh" === Just (Input ["abc"] [("ab", "cd"), ("ef", "gh")]) + readInput "%2fabc" === Nothing + readInput "%2F" === Nothing + readInput "def%2fabc" === Nothing + readInput "." === Nothing + readInput ".." === Nothing + readInput "..a" === Just (Input ["..a"] []) + readInput "../a" === Nothing + readInput "a/../a" === Nothing + readInput "%2e" === Nothing + readInput "%2E" === Nothing