tensorflow-logging (empty) → 0.1.0.0
raw patch · 5 files changed
+496/−0 lines, 5 filesdep +HUnitdep +basedep +bytestringsetup-changed
Dependencies added: HUnit, base, bytestring, conduit, data-default, directory, exceptions, filepath, hostname, lens-family, proto-lens, resourcet, stm, stm-chans, stm-conduit, temporary, tensorflow, tensorflow-core-ops, tensorflow-logging, tensorflow-ops, tensorflow-proto, tensorflow-records-conduit, test-framework, test-framework-hunit, text, time, transformers
Files
- LICENSE +203/−0
- Setup.hs +3/−0
- src/TensorFlow/Logging.hs +159/−0
- tensorflow-logging.cabal +67/−0
- tests/LoggingTest.hs +64/−0
+ LICENSE view
@@ -0,0 +1,203 @@+Copyright 2016 The TensorFlow Authors. All rights reserved.++ Apache License+ Version 2.0, January 2004+ http://www.apache.org/licenses/++ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++ 1. Definitions.++ "License" shall mean the terms and conditions for use, reproduction,+ and distribution as defined by Sections 1 through 9 of this document.++ "Licensor" shall mean the copyright owner or entity authorized by+ the copyright owner that is granting the License.++ "Legal Entity" shall mean the union of the acting entity and all+ other entities that control, are controlled by, or are under common+ control with that entity. For the purposes of this definition,+ "control" means (i) the power, direct or indirect, to cause the+ direction or management of such entity, whether by contract or+ otherwise, or (ii) ownership of fifty percent (50%) or more of the+ outstanding shares, or (iii) beneficial ownership of such entity.++ "You" (or "Your") shall mean an individual or Legal Entity+ exercising permissions granted by this License.++ "Source" form shall mean the preferred form for making modifications,+ including but not limited to software source code, documentation+ source, and configuration files.++ "Object" form shall mean any form resulting from mechanical+ transformation or translation of a Source form, including but+ not limited to compiled object code, generated documentation,+ and conversions to other media types.++ "Work" shall mean the work of authorship, whether in Source or+ Object form, made available under the License, as indicated by a+ copyright notice that is included in or attached to the work+ (an example is provided in the Appendix below).++ "Derivative Works" shall mean any work, whether in Source or Object+ form, that is based on (or derived from) the Work and for which the+ editorial revisions, annotations, elaborations, or other modifications+ represent, as a whole, an original work of authorship. For the purposes+ of this License, Derivative Works shall not include works that remain+ separable from, or merely link (or bind by name) to the interfaces of,+ the Work and Derivative Works thereof.++ "Contribution" shall mean any work of authorship, including+ the original version of the Work and any modifications or additions+ to that Work or Derivative Works thereof, that is intentionally+ submitted to Licensor for inclusion in the Work by the copyright owner+ or by an individual or Legal Entity authorized to submit on behalf of+ the copyright owner. For the purposes of this definition, "submitted"+ means any form of electronic, verbal, or written communication sent+ to the Licensor or its representatives, including but not limited to+ communication on electronic mailing lists, source code control systems,+ and issue tracking systems that are managed by, or on behalf of, the+ Licensor for the purpose of discussing and improving the Work, but+ excluding communication that is conspicuously marked or otherwise+ designated in writing by the copyright owner as "Not a Contribution."++ "Contributor" shall mean Licensor and any individual or Legal Entity+ on behalf of whom a Contribution has been received by Licensor and+ subsequently incorporated within the Work.++ 2. Grant of Copyright License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ copyright license to reproduce, prepare Derivative Works of,+ publicly display, publicly perform, sublicense, and distribute the+ Work and such Derivative Works in Source or Object form.++ 3. Grant of Patent License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ (except as stated in this section) patent license to make, have made,+ use, offer to sell, sell, import, and otherwise transfer the Work,+ where such license applies only to those patent claims licensable+ by such Contributor that are necessarily infringed by their+ Contribution(s) alone or by combination of their Contribution(s)+ with the Work to which such Contribution(s) was submitted. If You+ institute patent litigation against any entity (including a+ cross-claim or counterclaim in a lawsuit) alleging that the Work+ or a Contribution incorporated within the Work constitutes direct+ or contributory patent infringement, then any patent licenses+ granted to You under this License for that Work shall terminate+ as of the date such litigation is filed.++ 4. Redistribution. You may reproduce and distribute copies of the+ Work or Derivative Works thereof in any medium, with or without+ modifications, and in Source or Object form, provided that You+ meet the following conditions:++ (a) You must give any other recipients of the Work or+ Derivative Works a copy of this License; and++ (b) You must cause any modified files to carry prominent notices+ stating that You changed the files; and++ (c) You must retain, in the Source form of any Derivative Works+ that You distribute, all copyright, patent, trademark, and+ attribution notices from the Source form of the Work,+ excluding those notices that do not pertain to any part of+ the Derivative Works; and++ (d) If the Work includes a "NOTICE" text file as part of its+ distribution, then any Derivative Works that You distribute must+ include a readable copy of the attribution notices contained+ within such NOTICE file, excluding those notices that do not+ pertain to any part of the Derivative Works, in at least one+ of the following places: within a NOTICE text file distributed+ as part of the Derivative Works; within the Source form or+ documentation, if provided along with the Derivative Works; or,+ within a display generated by the Derivative Works, if and+ wherever such third-party notices normally appear. The contents+ of the NOTICE file are for informational purposes only and+ do not modify the License. You may add Your own attribution+ notices within Derivative Works that You distribute, alongside+ or as an addendum to the NOTICE text from the Work, provided+ that such additional attribution notices cannot be construed+ as modifying the License.++ You may add Your own copyright statement to Your modifications and+ may provide additional or different license terms and conditions+ for use, reproduction, or distribution of Your modifications, or+ for any such Derivative Works as a whole, provided Your use,+ reproduction, and distribution of the Work otherwise complies with+ the conditions stated in this License.++ 5. Submission of Contributions. Unless You explicitly state otherwise,+ any Contribution intentionally submitted for inclusion in the Work+ by You to the Licensor shall be under the terms and conditions of+ this License, without any additional terms or conditions.+ Notwithstanding the above, nothing herein shall supersede or modify+ the terms of any separate license agreement you may have executed+ with Licensor regarding such Contributions.++ 6. Trademarks. This License does not grant permission to use the trade+ names, trademarks, service marks, or product names of the Licensor,+ except as required for reasonable and customary use in describing the+ origin of the Work and reproducing the content of the NOTICE file.++ 7. Disclaimer of Warranty. Unless required by applicable law or+ agreed to in writing, Licensor provides the Work (and each+ Contributor provides its Contributions) on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+ implied, including, without limitation, any warranties or conditions+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+ PARTICULAR PURPOSE. You are solely responsible for determining the+ appropriateness of using or redistributing the Work and assume any+ risks associated with Your exercise of permissions under this License.++ 8. Limitation of Liability. In no event and under no legal theory,+ whether in tort (including negligence), contract, or otherwise,+ unless required by applicable law (such as deliberate and grossly+ negligent acts) or agreed to in writing, shall any Contributor be+ liable to You for damages, including any direct, indirect, special,+ incidental, or consequential damages of any character arising as a+ result of this License or out of the use or inability to use the+ Work (including but not limited to damages for loss of goodwill,+ work stoppage, computer failure or malfunction, or any and all+ other commercial damages or losses), even if such Contributor+ has been advised of the possibility of such damages.++ 9. Accepting Warranty or Additional Liability. While redistributing+ the Work or Derivative Works thereof, You may choose to offer,+ and charge a fee for, acceptance of support, warranty, indemnity,+ or other liability obligations and/or rights consistent with this+ License. However, in accepting such obligations, You may act only+ on Your own behalf and on Your sole responsibility, not on behalf+ of any other Contributor, and only if You agree to indemnify,+ defend, and hold each Contributor harmless for any liability+ incurred by, or claims asserted against, such Contributor by reason+ of your accepting any such warranty or additional liability.++ END OF TERMS AND CONDITIONS++ APPENDIX: How to apply the Apache License to your work.++ To apply the Apache License to your work, attach the following+ boilerplate notice, with the fields enclosed by brackets "[]"+ replaced with your own identifying information. (Don't include+ the brackets!) The text should be enclosed in the appropriate+ comment syntax for the file format. We also recommend that a+ file or class name and description of purpose be included on the+ same "printed page" as the copyright notice for easier+ identification within third-party archives.++ Copyright 2016, The TensorFlow Authors.++ Licensed under the Apache License, Version 2.0 (the "License");+ you may not use this file except in compliance with the License.+ You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.
+ Setup.hs view
@@ -0,0 +1,3 @@+import Distribution.Simple++main = defaultMain
+ src/TensorFlow/Logging.hs view
@@ -0,0 +1,159 @@+-- Copyright 2016 TensorFlow authors.+--+-- Licensed under the Apache License, Version 2.0 (the "License");+-- you may not use this file except in compliance with the License.+-- You may obtain a copy of the License at+--+-- http://www.apache.org/licenses/LICENSE-2.0+--+-- Unless required by applicable law or agreed to in writing, software+-- distributed under the License is distributed on an "AS IS" BASIS,+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+-- See the License for the specific language governing permissions and+-- limitations under the License.++-- | TensorBoard Summary generation. Provides type safe wrappers around raw+-- string emitting CoreOps.+--+-- Example use:+--+-- > -- Call summary functions while constructing the graph.+-- > createModel = do+-- > loss <- -- ...+-- > TF.scalarSummary loss+-- >+-- > -- Write summaries to an EventWriter.+-- > train = TF.withEventWriter "/path/to/logs" $ \eventWriter -> do+-- > summaryTensor <- TF.build TF.allSummaries+-- > forM_ [1..] $ \step -> do+-- > if (step % 100 == 0)+-- > then do+-- > ((), summaryBytes) <- TF.run (trainStep, summaryTensor)+-- > let summary = decodeMessageOrDie (TF.unScalar summaryBytes)+-- > TF.logSummary eventWriter step summary+-- > else TF.run_ trainStep++{-# LANGUAGE TypeOperators #-}++module TensorFlow.Logging+ ( EventWriter+ , withEventWriter+ , logEvent+ , logSummary+ , SummaryTensor+ , histogramSummary+ , scalarSummary+ , mergeAllSummaries+ ) where++import Control.Concurrent (forkFinally)+import Control.Concurrent.MVar (MVar, newEmptyMVar, readMVar, putMVar)+import Control.Concurrent.STM (atomically)+import Control.Concurrent.STM.TBMQueue (TBMQueue, newTBMQueueIO, closeTBMQueue, writeTBMQueue)+import Control.Monad.Catch (MonadMask, bracket)+import Control.Monad.IO.Class (MonadIO, liftIO)+import Control.Monad.Trans.Resource (runResourceT)+import Data.ByteString (ByteString)+import Data.Conduit ((=$=))+import Data.Conduit.TQueue (sourceTBMQueue)+import Data.Default (def)+import Data.Int (Int64)+import Data.ProtoLens (encodeMessage)+import Data.Time.Clock (getCurrentTime)+import Data.Time.Clock.POSIX (utcTimeToPOSIXSeconds)+import Lens.Family2 ((.~), (&))+import Network.HostName (getHostName)+import Proto.Tensorflow.Core.Framework.Summary (Summary)+import Proto.Tensorflow.Core.Util.Event (Event, fileVersion, step, summary, wallTime)+import System.Directory (createDirectoryIfMissing)+import System.FilePath ((</>))+import TensorFlow.Build (MonadBuild)+import TensorFlow.Ops (scalar)+import TensorFlow.Records.Conduit (sinkTFRecords)+import TensorFlow.Tensor (Tensor, render, SummaryTensor, addSummary, collectAllSummaries)+import TensorFlow.Types (TensorType, type(/=))+import Text.Printf (printf)+import qualified Data.ByteString.Lazy as L+import qualified Data.Conduit as Conduit+import qualified Data.Conduit.List as Conduit+import qualified Data.Text as T+import qualified TensorFlow.GenOps.Core as CoreOps++-- | Handle for logging TensorBoard events safely from multiple threads.+data EventWriter = EventWriter (TBMQueue Event) (MVar ())++-- | Writes Event protocol buffers to event files.+withEventWriter ::+ (MonadIO m, MonadMask m)+ => FilePath+ -- ^ logdir. Local filesystem directory where event file will be written.+ -> (EventWriter -> m a)+ -> m a+withEventWriter logdir =+ bracket (liftIO (newEventWriter logdir)) (liftIO . closeEventWriter)++newEventWriter :: FilePath -> IO EventWriter+newEventWriter logdir = do+ createDirectoryIfMissing True logdir+ t <- doubleWallTime+ hostname <- getHostName+ let filename = printf (logdir </> "events.out.tfevents.%010d.%s")+ (truncate t :: Integer) hostname+ -- Asynchronously consume events from a queue.+ -- We use a bounded queue to ensure the producer doesn't get too far ahead+ -- of the consumer. The buffer size was picked arbitrarily.+ q <- newTBMQueueIO 1024+ -- Use an MVar to signal that the worker thread has completed.+ done <- newEmptyMVar+ let writer = EventWriter q done+ consumeQueue = runResourceT $ Conduit.runConduit $+ sourceTBMQueue q+ =$= Conduit.map (L.fromStrict . encodeMessage)+ =$= sinkTFRecords filename+ _ <- forkFinally consumeQueue (\_ -> putMVar done ())+ logEvent writer $ def & wallTime .~ t+ & fileVersion .~ T.pack "brain.Event:2"+ return writer++closeEventWriter :: EventWriter -> IO ()+closeEventWriter (EventWriter q done) =+ atomically (closeTBMQueue q) >> readMVar done++-- | Logs the given Event protocol buffer.+logEvent :: MonadIO m => EventWriter -> Event -> m ()+logEvent (EventWriter q _) pb = liftIO (atomically (writeTBMQueue q pb))++-- | Logs the given Summary event with an optional global step (use 0 if not+-- applicable).+logSummary :: MonadIO m => EventWriter -> Int64 -> Summary -> m ()+logSummary writer step' summaryProto = do+ t <- liftIO doubleWallTime+ logEvent writer (def & wallTime .~ t+ & step .~ step'+ & summary .~ summaryProto+ )++-- Number of seconds since epoch.+doubleWallTime :: IO Double+doubleWallTime = asDouble <$> getCurrentTime+ where asDouble t = fromRational (toRational (utcTimeToPOSIXSeconds t))++-- | Adds a 'CoreOps.histogramSummary' node. The tag argument is intentionally+-- limited to a single value for simplicity.+histogramSummary ::+ (MonadBuild m, TensorType t, t /= ByteString, t /= Bool)+ -- OneOf '[Int16, Int32, Int64, Int8, Word16, Word8, Double, Float] t)+ => ByteString -> Tensor v t -> m ()+histogramSummary tag = addSummary . CoreOps.histogramSummary (scalar tag)++-- | Adds a 'CoreOps.scalarSummary' node.+scalarSummary ::+ (TensorType t, t /= ByteString, t /= Bool, MonadBuild m)+ -- (TensorType t,+ -- OneOf '[Int16, Int32, Int64, Int8, Word16, Word8, Double, Float] t)+ => ByteString -> Tensor v t -> m ()+scalarSummary tag = addSummary . CoreOps.scalarSummary (scalar tag)++-- | Merge all summaries accumulated in the 'Build' into one summary.+mergeAllSummaries :: MonadBuild m => m SummaryTensor+mergeAllSummaries = collectAllSummaries >>= render . CoreOps.mergeSummary
+ tensorflow-logging.cabal view
@@ -0,0 +1,67 @@+name: tensorflow-logging+version: 0.1.0.0+synopsis: TensorBoard related functionality.+description: Please see README.md+homepage: https://github.com/tensorflow/haskell#readme+license: Apache+license-file: LICENSE+author: TensorFlow authors+maintainer: tensorflow-haskell@googlegroups.com+copyright: Google Inc.+category: Machine Learning+build-type: Simple+cabal-version: >=1.22++library+ hs-source-dirs: src+ exposed-modules: TensorFlow.Logging+ build-depends: base >= 4.7 && < 5+ , bytestring+ , conduit+ , data-default+ , directory+ , exceptions+ , filepath+ , hostname+ , lens-family+ , proto-lens == 0.2.*+ , resourcet+ , stm+ , stm-chans+ , stm-conduit+ , tensorflow == 0.1.*+ , tensorflow-core-ops == 0.1.*+ , tensorflow-ops == 0.1.*+ , tensorflow-proto == 0.1.*+ , tensorflow-records-conduit+ , text+ , time+ , transformers+ default-language: Haskell2010++Test-Suite LoggingTest+ default-language: Haskell2010+ type: exitcode-stdio-1.0+ main-is: LoggingTest.hs+ hs-source-dirs: tests+ build-depends: HUnit+ , base+ , bytestring+ , conduit+ , data-default+ , directory+ , filepath+ , lens-family+ , proto-lens+ , resourcet+ , temporary+ , tensorflow-logging+ , tensorflow-proto+ , tensorflow-records-conduit+ , test-framework+ , test-framework-hunit+ , text++source-repository head+ type: git+ location: https://github.com/tensorflow/haskell
+ tests/LoggingTest.hs view
@@ -0,0 +1,64 @@+-- Copyright 2016 TensorFlow authors.+--+-- Licensed under the Apache License, Version 2.0 (the "License");+-- you may not use this file except in compliance with the License.+-- You may obtain a copy of the License at+--+-- http://www.apache.org/licenses/LICENSE-2.0+--+-- Unless required by applicable law or agreed to in writing, software+-- distributed under the License is distributed on an "AS IS" BASIS,+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+-- See the License for the specific language governing permissions and+-- limitations under the License.++-- | Tests for TensorFlow.Logging.+module Main where++import Control.Monad.Trans.Resource (runResourceT)+import Data.Conduit ((=$=))+import Data.Default (def)+import Data.List ((\\))+import Data.ProtoLens (decodeMessageOrDie)+import Lens.Family2 ((^.), (.~), (&))+import Proto.Tensorflow.Core.Util.Event (Event, fileVersion, step)+import System.Directory (getDirectoryContents)+import System.FilePath ((</>))+import System.IO.Temp (withSystemTempDirectory)+import TensorFlow.Logging (withEventWriter, logEvent)+import TensorFlow.Records.Conduit (sourceTFRecords)+import Test.Framework (defaultMain, Test)+import Test.Framework.Providers.HUnit (testCase)+import Test.HUnit (assertBool, assertEqual)+import qualified Data.ByteString.Lazy as BL+import qualified Data.Conduit as Conduit+import qualified Data.Conduit.List as Conduit+import qualified Data.Text as T++-- TODO: This has been added to System.Directory in newer versions.+listDirectory :: String -> IO [String]+listDirectory dir = (\\ [".", ".."]) <$> getDirectoryContents dir++testEventWriter :: Test+testEventWriter = testCase "EventWriter" $+ withSystemTempDirectory "event_writer_logs" $ \dir -> do+ assertEqual "No file before" [] =<< listDirectory dir+ let expected = [ (def :: Event) & step .~ 10+ , def & step .~ 222+ , def & step .~ 8+ ]+ withEventWriter dir $ \eventWriter ->+ mapM_ (logEvent eventWriter) expected+ files <- listDirectory dir+ assertEqual "One file exists after" 1 (length files)+ records <- runResourceT $ Conduit.runConduit $+ sourceTFRecords (dir </> head files) =$= Conduit.consume+ assertBool "File is not empty" (not (null records))+ let (header:body) = decodeMessageOrDie . BL.toStrict <$> records+ assertEqual "Header has expected version"+ (T.pack "brain.Event:2") (header ^. fileVersion)+ assertEqual "Body has expected records" expected body++main :: IO ()+main = defaultMain [ testEventWriter+ ]