packages feed

shodan 0.1.0.0 → 0.2.0.0

raw patch · 8 files changed

+339/−10 lines, 8 filesnew-component:exe:shodan-searchPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

+ Shodan: InvalidPage :: ShodanError
+ Shodan: SearchMatch :: Text -> Int -> Maybe Text -> Maybe Text -> SearchMatch
+ Shodan: SearchResult :: Integer -> [SearchMatch] -> Map Text [Facet] -> SearchResult
+ Shodan: [matchAsn] :: SearchMatch -> Maybe Text
+ Shodan: [matchIp] :: SearchMatch -> Text
+ Shodan: [matchPort] :: SearchMatch -> Int
+ Shodan: [matchProduct] :: SearchMatch -> Maybe Text
+ Shodan: [searchFacets] :: SearchResult -> Map Text [Facet]
+ Shodan: [searchMatches] :: SearchResult -> [SearchMatch]
+ Shodan: [searchTotal] :: SearchResult -> Integer
+ Shodan: data SearchMatch
+ Shodan: data SearchResult
+ Shodan: instance Data.Aeson.Types.FromJSON.FromJSON Shodan.SearchMatch
+ Shodan: instance Data.Aeson.Types.FromJSON.FromJSON Shodan.SearchResult
+ Shodan: instance GHC.Classes.Eq Shodan.SearchMatch
+ Shodan: instance GHC.Classes.Eq Shodan.SearchResult
+ Shodan: instance GHC.Show.Show Shodan.SearchMatch
+ Shodan: instance GHC.Show.Show Shodan.SearchResult
+ Shodan: searchHosts :: Client -> Text -> Int -> IO (Either ShodanError SearchResult)

Files

+ CHANGELOG.md view
@@ -0,0 +1,36 @@+# Changelog++This package follows the [Haskell Package Versioning Policy](https://pvp.haskell.org/).++## 0.2.0.0 — Unreleased++### Breaking changes++- Added `InvalidPage` to the public `ShodanError` type. Consumers that explicitly+  handle every constructor should add a case for this error. This datatype+  change requires a major version increase under PVP.++### Additions++- Added `searchHosts` for one page of `GET /shodan/host/search`, with an explicit+  page number starting at 1. Invalid page numbers are rejected before HTTP;+  additional pages are not fetched automatically.+- Added `SearchMatch` with IP, port, optional product, and optional ASN fields.+- Added `SearchResult` with total results, the current page of service matches,+  and optional facet breakdowns.+- Added the `shodan-search` executable, with an optional page argument and+  tabular service output. It reads `SHODAN_API_KEY` from the environment.+- Added search JSON parsing and local HTTP tests for page selection, query+  encoding, invalid pages, error responses, and malformed responses.+- Updated the README and Haddock documentation with search usage and credit+  behavior. Live search was also verified manually; automated tests remain local.++## 0.1.0.0 — 2026-11-20++- Initial Hackage release under the BSD 3-Clause License.+- Added API plan information and search result count requests.+- Added `ApiInfo`, `UsageLimits`, `CountResult`, and `Facet` response types.+- Added TLS client construction, redacted client display, HTTP error handling,+  and a finite response timeout without automatic redirect following.+- Added the `shodan-api-info` and `shodan-count` executable examples, local+  response parsing and HTTP tests, and Haddock documentation.
README.md view
@@ -4,8 +4,12 @@  Licensed under the [BSD 3-Clause License](LICENSE). -This project currently supports `GET /api-info` (`ApiInfo` and `UsageLimits`)-and `GET /shodan/host/count` (`CountResult`). The API-info example fields and+Version **0.2.0.0** is being prepared for release. See the [changelog](CHANGELOG.md)+for additions and the `ShodanError` constructor change requiring a PVP major bump.++This project supports `GET /api-info` (`ApiInfo` and `UsageLimits`),+`GET /shodan/host/count` (`CountResult`), and one-page+`GET /shodan/host/search` (`SearchResult`). The API-info example fields and the count response's `total` field are required; unknown fields are ignored. Numeric values use `Integer`; count totals must be nonnegative. @@ -75,8 +79,9 @@ countHosts :: Client -> Text -> IO (Either ShodanError CountResult) ``` -Read the total with `countTotal`. Optional facet requests and individual search-results are not supported yet. According to the [official count endpoint+Read the total with `countTotal`. For individual search results, use+`searchHosts` as described below. Optional facet requests are not supported yet.+According to the [official count endpoint documentation](https://developer.shodan.io/api), this call uses no query credits.  ## Facet response type@@ -108,6 +113,49 @@ Both fields are required. Unknown fields are ignored, while missing fields, invalid counts, and non-object buckets are rejected. +## Search (0.2.0.0, unreleased)++Version 0.2.0.0 adds one-page searches:++```haskell+searchHosts :: Client -> Text -> Int -> IO (Either ShodanError SearchResult)+```++The page number starts at 1. Values below 1 return `Left InvalidPage` before+any request. Each call retrieves exactly the requested page; additional pages+are not fetched automatically. Shodan requires pages past 100 to be requested+in order.++With `SHODAN_API_KEY` set, the executable accepts a query and an optional page+(default 1):++```sh+cabal run shodan-search -- 'product:nginx country:KR'+cabal run shodan-search -- 'product:nginx country:KR' 2+```++These commands call Shodan and can use query credits: a filtered query or any+page after the first uses a credit under the [official API+policy](https://developer.shodan.io/api). The example prints the total across+all pages, the selected page number, the returned service count, and a table of+IP, port, product, and ASN. Missing product/ASN values are displayed as `-`.++`SearchMatch` describes one matching service with `matchIp`, `matchPort`,+`matchProduct`, and `matchAsn`. IP text and an integer port from 0 through 65535+are required. Product and ASN are optional: missing or null values become+`Nothing`, while other non-text values are rejected. IP syntax is not validated.+Unknown banner fields are ignored.++`SearchResult` contains `searchTotal`, `searchMatches`, and `searchFacets`.+The total covers all pages, while the matches describe one page and preserve+service order (multiple services may share an IP). Total and matches are+required; the total is a nonnegative `Integer`. Facets follow the same parsing+rules as `CountResult`.++Optional facet requests and field selection are not supported yet. Search+tests use local JSON fixtures and a mock HTTP server, and do not consume Shodan+query credits. A one-page live search has also succeeded manually.+ ## Build  The initial development compiler is GHC 9.8.4.@@ -136,8 +184,9 @@ With the Cabal executable directory on `PATH`, run from the project directory:  ```sh-stylish-haskell -i src/Shodan.hs test/Main.hs examples/api-info/Main.hs+stylish-haskell -i src/Shodan.hs test/*.hs examples/api-info/Main.hs stylish-haskell -i examples/count/Main.hs+stylish-haskell -i examples/search/Main.hs cabal test ``` @@ -145,10 +194,13 @@  - `shodan.cabal`: package and library settings. - `cabal.project`: local Cabal project settings.+- `CHANGELOG.md`: release history and compatibility notes. - `src/Shodan.hs`: client, errors, response records, and JSON parsing. - `examples/api-info/Main.hs`: executable example using an environment variable. - `examples/count/Main.hs`: count-query executable using the same environment variable.+- `examples/search/Main.hs`: one-page search executable with an optional page argument. - `test/Main.hs`: response parsing and HTTP tests against a local mock server.+- `test/SearchTests.hs`: search-response JSON parsing tests.  No API key is needed to build or test. Automated tests do not call Shodan. They check request encoding, successful responses, HTTP errors, invalid JSON,
examples/api-info/Main.hs view
@@ -28,3 +28,4 @@ errorMessage NetworkError = "Could not reach Shodan (connection failure or timeout)." errorMessage (HttpError status) = "Shodan returned HTTP " ++ show status ++ "." errorMessage InvalidResponse = "Shodan returned an unexpected response."+errorMessage InvalidPage = "Search page numbers must be at least 1."
+ examples/search/Main.hs view
@@ -0,0 +1,48 @@+module Main (main) where++import           Data.Char          (isSpace)+import           Data.List          (intercalate)+import qualified Data.Text          as Text+import           Shodan             (SearchMatch (..), SearchResult (..),+                                     newClient, searchHosts)+import           System.Environment (getArgs, lookupEnv)+import           System.Exit        (die)+import           Text.Read          (readMaybe)++main :: IO ()+main = do+  arguments <- getArgs+  (query, page) <- parseArguments arguments+  configuredKey <- lookupEnv "SHODAN_API_KEY"+  case configuredKey of+    Nothing -> die "Set the SHODAN_API_KEY environment variable before running."+    Just key+      | all isSpace key -> die "SHODAN_API_KEY must not be empty or whitespace."+      | otherwise -> do+          client <- newClient (Text.pack key)+          result <- searchHosts client query page+          case result of+            Left err -> die ("Shodan search failed: " ++ show err)+            Right found -> do+              putStrLn ("Total: " ++ show (searchTotal found))+              putStrLn ("Page: " ++ show page)+              putStrLn ("Returned: " ++ show (length (searchMatches found)))+              putStrLn "IP\tPORT\tPRODUCT\tASN"+              mapM_ printMatch (searchMatches found)++parseArguments :: [String] -> IO (Text.Text, Int)+parseArguments [query] = parseArguments [query, "1"]+parseArguments [query, pageText]+  | not (all isSpace query)+  , Just page <- (readMaybe pageText :: Maybe Integer)+  , page >= 1+  , page <= toInteger (maxBound :: Int) = pure (Text.pack query, fromInteger page)+parseArguments _ = die "Usage: shodan-search 'QUERY' [PAGE >= 1]"++printMatch :: SearchMatch -> IO ()+printMatch match = putStrLn (intercalate "\t"+  [ Text.unpack (matchIp match)+  , show (matchPort match)+  , maybe "-" Text.unpack (matchProduct match)+  , maybe "-" Text.unpack (matchAsn match)+  ])
shodan.cabal view
@@ -1,13 +1,14 @@ cabal-version: 2.4 name: shodan-version: 0.1.0.0+version: 0.2.0.0 license: BSD-3-Clause license-file: LICENSE synopsis: A simple Haskell client for the Shodan API description:   A small Shodan API client using ordinary records and explicit IO functions.-  Supports API plan information and search result counts, with JSON response-  types for facet statistics. Includes executable examples and local HTTP tests.+  Supports API plan information, search result counts, and paginated searches,+  with JSON response types for facet statistics. Includes executable examples+  and local HTTP tests. category: Network author: Joonkyu Park maintainer: vpark45@gmail.com@@ -15,7 +16,7 @@ bug-reports: https://github.com/nattybear/shodan/issues tested-with: GHC == 9.8.4 build-type: Simple-extra-source-files: README.md, .stylish-haskell.yaml+extra-source-files: README.md, CHANGELOG.md, .stylish-haskell.yaml  source-repository head   type: git@@ -55,9 +56,20 @@   default-language: Haskell2010   ghc-options: -Wall -threaded +executable shodan-search+  main-is: Main.hs+  hs-source-dirs: examples/search+  build-depends:+    base >= 4.19 && < 5,+    shodan,+    text >= 2.1 && < 2.2+  default-language: Haskell2010+  ghc-options: -Wall -threaded+ test-suite shodan-test   type: exitcode-stdio-1.0   main-is: Main.hs+  other-modules: SearchTests   hs-source-dirs: test   build-depends:     base >= 4.19 && < 5,
src/Shodan.hs view
@@ -8,18 +8,22 @@   , ShodanError (..)   , getApiInfo   , countHosts+  , searchHosts   , CountResult (..)   , Facet (..)+  , SearchMatch (..)+  , SearchResult (..)   , ApiInfo (..)   , UsageLimits (..)   ) where  import           Control.Exception       (catch) import           Data.Aeson              (FromJSON (..), Value, eitherDecode',-                                          withObject, (.!=), (.:!), (.:))+                                          withObject, (.!=), (.:!), (.:), (.:?)) import           Data.Map.Strict         (Map) import qualified Data.Map.Strict         as Map import           Data.Text               (Text)+import qualified Data.Text               as DataText import qualified Data.Text.Encoding      as Text import qualified Network.HTTP.Client     as HTTP import           Network.HTTP.Client.TLS (tlsManagerSettings)@@ -49,6 +53,8 @@   = NetworkError   | HttpError Int   | InvalidResponse+  | InvalidPage+    -- ^ Search page numbers must be at least 1. No request was made.   deriving (Eq, Show)  -- | Retrieve @GET https://api.shodan.io/api-info@.@@ -65,6 +71,17 @@ countHosts :: Client -> Text -> IO (Either ShodanError CountResult) countHosts client query = getJSON client "/shodan/host/count" [("query", query)] +-- | Retrieve one page from @GET /shodan/host/search@.+-- The page number starts at 1. No additional pages are fetched automatically.+-- Filtered queries and requests after page 1 use query credits. Requests past+-- page 100 must be made in order, as required by Shodan. Optional facets and+-- field selection are not requested.+searchHosts :: Client -> Text -> Int -> IO (Either ShodanError SearchResult)+searchHosts client query page+  | page < 1 = pure (Left InvalidPage)+  | otherwise = getJSON client "/shodan/host/search"+      [("query", query), ("page", DataText.pack (show page))]+ -- | Total matching results and optional breakdowns from @GET /shodan/host/count@. data CountResult = CountResult   { countTotal  :: Integer@@ -102,6 +119,50 @@     if count < 0       then fail "Facet count must not be negative"       else pure (Facet value count)++-- | One matching service in a search response, rather than a unique host.+-- Unmodelled banner fields are ignored. The IP address is preserved as text;+-- its syntax is not validated here.+data SearchMatch = SearchMatch+  { matchIp      :: Text+    -- ^ Required @ip_str@ field, usable for IPv4 or IPv6 text.+  , matchPort    :: Int+    -- ^ Required integer port, from 0 through 65535.+  , matchProduct :: Maybe Text+    -- ^ Detected product; absent or null means 'Nothing'.+  , matchAsn     :: Maybe Text+    -- ^ ASN such as @AS65536@; absent or null means 'Nothing'.+  } deriving (Eq, Show)++instance FromJSON SearchMatch where+  parseJSON = withObject "SearchMatch" $ \o -> do+    ip <- o .: "ip_str"+    port <- o .: "port"+    productName <- o .:? "product"+    asn <- o .:? "asn"+    if port < 0 || port > 65535+      then fail "Search match port must be between 0 and 65535"+      else pure (SearchMatch ip port productName asn)++-- | One page returned by @GET /shodan/host/search@.+-- The total covers the whole query and may exceed this page's match count.+data SearchResult = SearchResult+  { searchTotal   :: Integer+    -- ^ Required nonnegative total across all pages.+  , searchMatches :: [SearchMatch]+    -- ^ Required service list; an empty page is valid. Order is preserved.+  , searchFacets  :: Map Text [Facet]+    -- ^ Optional breakdowns. Missing means an empty map; null is rejected.+  } deriving (Eq, Show)++instance FromJSON SearchResult where+  parseJSON = withObject "SearchResult" $ \o -> do+    total <- o .: "total"+    matches <- o .: "matches"+    facets <- o .:! "facets" .!= Map.empty+    if total < 0+      then fail "Search total must not be negative"+      else pure (SearchResult total matches facets)  -- Shared transport keeps status handling and key protection consistent. getJSON :: FromJSON a => Client -> Text -> [(Text, Text)]
test/Main.hs view
@@ -16,6 +16,7 @@ import qualified Network.HTTP.Types       as HTTPTypes import qualified Network.Wai              as Wai import           Network.Wai.Handler.Warp (testWithApplication)+import           SearchTests              (searchParsingTests) import           Shodan  main :: IO ()@@ -36,8 +37,10 @@   rejects "non-object response" (Array mempty)   facetTests   countParsingTests+  searchParsingTests   httpTests   countTests+  searchHttpTests   putStrLn "All response parsing and HTTP tests passed."  facetTests :: IO ()@@ -178,6 +181,48 @@     (Left (HttpError 401))   countResponse "count rate limited" HTTPTypes.status429 "not JSON"     (Left (HttpError 429))++searchHttpTests :: IO ()+searchHttpTests = do+  let key = "search-key+&"+      query = "http.title:\"한글 + & ?\" product:nginx"+      responseBody = "{\"total\":101,\"matches\":[{\"ip_str\":\"192.0.2.1\",\"port\":443,\"product\":\"nginx\",\"asn\":\"AS65536\"}]}"+      expected = Right (SearchResult 101+        [SearchMatch "192.0.2.1" 443 (Just "nginx") (Just "AS65536")] Map.empty)+      application page request respond = do+        check "search GET" (Wai.requestMethod request == "GET")+        check "search path" (Wai.pathInfo request == ["shodan", "host", "search"])+        check "search encoded parameters" (Wai.queryString request ==+          [("key", Just (TextEncoding.encodeUtf8 key)),+           ("query", Just (TextEncoding.encodeUtf8 query)),+           ("page", Just page)])+        respond (Wai.responseLBS HTTPTypes.status200 [] responseBody)+  withMockClient key (application "1") $ \client -> do+    actual <- searchHosts client query 1+    check "first search page" (actual == expected)+  withMockClient key (application "2") $ \client -> do+    actual <- searchHosts client query 2+    check "explicit second search page" (actual == expected)+  manager <- HTTP.newManager HTTP.defaultManagerSettings+    { HTTP.managerModifyRequest = \_ -> fail "Invalid page sent an HTTP request" }+  mapM_ (\page -> do+    actual <- searchHosts (newClientWithManager manager key) query page+    check "invalid page rejected before HTTP" (actual == Left InvalidPage)) [0, -1]+  let cases =+        [ ("empty search page", HTTPTypes.status200, "{\"total\":0,\"matches\":[]}",+            Right (SearchResult 0 [] Map.empty))+        , ("search unauthorized", HTTPTypes.status401, "{\"error\":\"search-key+&\"}",+            Left (HttpError 401))+        , ("search rate limited", HTTPTypes.status429, "not JSON", Left (HttpError 429))+        , ("search invalid JSON", HTTPTypes.status200, "not JSON", Left InvalidResponse)+        , ("search invalid match", HTTPTypes.status200,+            "{\"total\":1,\"matches\":[{}]}", Left InvalidResponse)+        ]+  mapM_ (\(label, status, body, wanted) -> do+    let mock _ respond = respond (Wai.responseLBS status [] body)+    withMockClient key mock $ \client -> do+      actual <- searchHosts client query 1+      check label (actual == wanted)) cases  countResponse :: String -> HTTPTypes.Status -> Lazy.ByteString   -> Either ShodanError CountResult -> IO ()
+ test/SearchTests.hs view
@@ -0,0 +1,74 @@+{-# LANGUAGE OverloadedStrings #-}++module SearchTests (searchParsingTests) where++import           Control.Monad        (unless)+import           Data.Aeson           (Value (..), eitherDecode')+import qualified Data.ByteString.Lazy as Lazy+import qualified Data.Map.Strict      as Map+import           Shodan++searchParsingTests :: IO ()+searchParsingTests = do+  -- Selected fields from the official GET /shodan/host/search example.+  -- Source: https://developer.shodan.io/api+  let match = SearchMatch "96.93.212.27" 443 (Just "nginx") (Just "AS7922")+      expected = SearchResult 23047224 [match]+        (Map.singleton "country" [Facet (String "US") 7883733])+      example = "{\"total\":23047224,\"matches\":[{\"ip_str\":\"96.93.212.27\",\"port\":443,\"product\":\"nginx\",\"asn\":\"AS7922\",\"http\":{\"server\":\"nginx\"}}],\"facets\":{\"country\":[{\"value\":\"US\",\"count\":7883733}]}}"+  check "documented search fields" (eitherDecode' example == Right expected)+  check "empty search" (eitherDecode' "{\"total\":0,\"matches\":[]}" ==+    Right (SearchResult 0 [] Map.empty))+  check "total can exceed page size" (eitherDecode'+    "{\"total\":100000000000000000000,\"matches\":[],\"future\":true}" ==+    Right (SearchResult 100000000000000000000 [] Map.empty))+  let minimal = SearchMatch "2001:db8::1" 0 Nothing Nothing+  check "IPv6 and absent optional fields" (eitherDecode'+    "{\"ip_str\":\"2001:db8::1\",\"port\":0}" == Right minimal)+  check "null optional fields and maximum port" (eitherDecode'+    "{\"ip_str\":\"192.0.2.1\",\"port\":65535,\"product\":null,\"asn\":null}" ==+    Right (SearchMatch "192.0.2.1" 65535 Nothing Nothing))+  let twoMatches = "{\"total\":2,\"matches\":[{\"ip_str\":\"192.0.2.1\",\"port\":80},{\"ip_str\":\"192.0.2.1\",\"port\":443}],\"facets\":{}}"+  check "multiple services on one IP retain order" (eitherDecode' twoMatches ==+    Right (SearchResult 2+      [SearchMatch "192.0.2.1" 80 Nothing Nothing,+       SearchMatch "192.0.2.1" 443 Nothing Nothing] Map.empty))+  mapM_ rejectsMatch+    [ ("missing IP", "{\"port\":443}")+    , ("null IP", "{\"ip_str\":null,\"port\":443}")+    , ("numeric IP", "{\"ip_str\":123,\"port\":443}")+    , ("missing port", "{\"ip_str\":\"192.0.2.1\"}")+    , ("null port", "{\"ip_str\":\"192.0.2.1\",\"port\":null}")+    , ("fractional port", "{\"ip_str\":\"192.0.2.1\",\"port\":1.5}")+    , ("negative port", "{\"ip_str\":\"192.0.2.1\",\"port\":-1}")+    , ("out-of-range port", "{\"ip_str\":\"192.0.2.1\",\"port\":65536}")+    , ("wrong product type", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"product\":true}")+    , ("wrong ASN type", "{\"ip_str\":\"192.0.2.1\",\"port\":443,\"asn\":65536}")+    , ("non-object match", "[]")+    ]+  mapM_ rejectsSearch+    [ ("missing total", "{\"matches\":[]}")+    , ("negative total", "{\"total\":-1,\"matches\":[]}")+    , ("fractional total", "{\"total\":1.5,\"matches\":[]}")+    , ("missing matches", "{\"total\":0}")+    , ("null matches", "{\"total\":0,\"matches\":null}")+    , ("object matches", "{\"total\":0,\"matches\":{}}")+    , ("invalid nested match", "{\"total\":1,\"matches\":[{}]}")+    , ("null facets", "{\"total\":0,\"matches\":[],\"facets\":null}")+    , ("invalid nested facet", "{\"total\":0,\"matches\":[],\"facets\":{\"country\":[{}]}}")+    , ("non-object search", "[]")+    , ("malformed JSON", "not JSON")+    ]++check :: String -> Bool -> IO ()+check label success = unless success (fail label)++rejectsMatch :: (String, Lazy.ByteString) -> IO ()+rejectsMatch (label, body) = case eitherDecode' body :: Either String SearchMatch of+  Left _  -> pure ()+  Right _ -> fail (label ++ ": unexpectedly accepted")++rejectsSearch :: (String, Lazy.ByteString) -> IO ()+rejectsSearch (label, body) = case eitherDecode' body :: Either String SearchResult of+  Left _  -> pure ()+  Right _ -> fail (label ++ ": unexpectedly accepted")