Large Image File Incomplete Loading

[ChasTheSpaz]

Only problem noted for my application (that searches for and displays image files), is incomplete loading of large (beyond 1k x 1k pixels, often beyond 2000 x 3000 pixel, as large as 6000 pixels on one axis) images into a new browser tab. Happens on Safari, Firefox and Chrome. The relevant output/diagnostic message from the run is currently delayed by one image load:

[22/Aug/2017:14:40:52 +1200] Exception leaked to httpSession during phase 'user handler':

During processing of request from 127.0.0.1:63718
request:
"GET /static/Users/charlesschatz/Desktop/Haskell/Threepenny-gui/static/Users/charlesschatz/Desktop/Classical%20Objects%20and%20Scenes/kottabos6.jpg HTTP/1.1\naccept-language: en-us\ndnt: 1\nreferer: http://localhost:8023/\nuser-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2 Safari/603.3.8\naccept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8\nupgrade-insecure-requests: 1\nconnection: keep-alive\ncookie: _ga=GA1.1.145967290.1481489723\naccept-encoding: gzip, deflate\nhost: localhost:8023\n\nsn="localhost:8023" c=127.0.0.1:63718 s=127.0.0.1:8023 ctx=/ clen=n/a\ncookies: Cookie {cookieName = "_ga", cookieValue = "GA1.1.145967290.1481489723", cookieExpires = Nothing, cookieDomain = Nothing, cookiePath = Nothing, cookieSecure = False, cookieHttpOnly = False}"
A web handler threw an exception. Details:
TerminateSessionException sendfile: invalid argument (Socket is not connected)
127.0.0.1 - - [22/Aug/2017:14:41:08 +1200] "GET /static/Users/charlesschatz/Desktop/Haskell/Threepenny-gui/static/Users/charlesschatz/Desktop/Mary&Greg%20Images/marygreg13.jpg HTTP/1.1" 304 0 "http://localhost:8023/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2 Safari/603.3.8"

I have occasionally seen the problem on smaller files. Also played with compile-time heap size to no avail, assuming it was a memory problem or perhaps garbage-collection issue.

[HeinrichApfelmus]

Most likely, the problem needs to be solved within the Snap web framework. Large files tend to give problems occasionally, see snapframework/snap-core#91 . It seems that the choice of browser and operating system matters.

[ChasTheSpaz]

Got the source to snap-core via cabal, changed the Setup.hs file to set the portable flag and also forced the portable flag during build during re-installation of the package. The code seems much faster as well as trouble-free for big files, but haven't really thrashed it yet. I have thus far mildly tested it on a recent Mac machine with Safari. Will return with report of tests on other browsers.

Many thanks for your help, Heinrich!

[ChasTheSpaz]

Further testing using Safari shows a peculiar display bug on one image (image shows fully and normally on a dedicated Safari tab loaded with File Open, but Threepenny server returns a decode-damaged image) and the usual incomplete loading of other large-ish jpg files. These conditions also existed prior to the snap-core update. However, on FireFox, some problematic jpgs won't load at all (implying errors in some of the images, others display just fine). It's a mixed bag of performance across browsers on a single OS. My application is technically straightforward, but these performance issues put a damper on releasing the code in a University environment.

[HeinrichApfelmus]

To discern whether the problem is with Threepenny or with Snap, you could try to write a minimal example that serves a large image (with serveFile or serveFileAs) using just the Snap framework. See https://gist.github.com/HeinrichApfelmus/5591688 .

[ChasTheSpaz]

Took a copy of SnapFails.hs, modifed it and tried to compile it - compile fails on the 'route' call:

ghc -v snap-file-fail.hs 
Glasgow Haskell Compiler, Version 8.0.2, stage 2 booted by GHC version 7.10.3
Using binary package database: /Library/Frameworks/GHC.framework/Versions/8.0.2-x86_64/usr/lib/ghc-8.0.2/package.conf.d/package.cache
Using binary package database: /Users/charlesschatz/.ghc/x86_64-darwin-8.0.2/package.conf.d/package.cache
loading package database /Library/Frameworks/GHC.framework/Versions/8.0.2-x86_64/usr/lib/ghc-8.0.2/package.conf.d
loading package database /Users/charlesschatz/.ghc/x86_64-darwin-8.0.2/package.conf.d
wired-in package ghc-prim mapped to ghc-prim-0.5.0.0
wired-in package integer-gmp mapped to integer-gmp-1.0.0.1
wired-in package base mapped to base-4.9.1.0
wired-in package rts mapped to rts
wired-in package template-haskell mapped to template-haskell-2.11.1.0
wired-in package ghc mapped to ghc-8.0.2
wired-in package dph-seq not found.
wired-in package dph-par not found.
Hsc static flags: 
loading package database /Library/Frameworks/GHC.framework/Versions/8.0.2-x86_64/usr/lib/ghc-8.0.2/package.conf.d
loading package database /Users/charlesschatz/.ghc/x86_64-darwin-8.0.2/package.conf.d
wired-in package ghc-prim mapped to ghc-prim-0.5.0.0
wired-in package integer-gmp mapped to integer-gmp-1.0.0.1
wired-in package base mapped to base-4.9.1.0
wired-in package rts mapped to rts-1.0
wired-in package template-haskell mapped to template-haskell-2.11.1.0
wired-in package ghc mapped to ghc-8.0.2
wired-in package dph-seq not found.
wired-in package dph-par not found.
*** Chasing dependencies:
Chasing modules from: *snap-file-fail.hs
!!! Chasing dependencies: finished in 2.78 milliseconds, allocated 1.123 megabytes
Stable obj: []
Stable BCO: []
Ready for upsweep
  [NONREC
      ModSummary {
         ms_hs_date = 2017-08-26 02:02:50 UTC
         ms_mod = Main,
         ms_textual_imps = [(Nothing, Prelude),
                            (Nothing, Snap.Internal.Routing), (Nothing, Snap.Util.FileServe),
                            (Nothing, Snap.Http.Server), (Nothing, Snap.Core),
                            (Nothing, System.Posix.Files), (Nothing, Data.Functor)]
         ms_srcimps = []
      }]
*** Deleting temp files:
Deleting: 
compile: input file snap-file-fail.hs
*** Checking old interface for Main:
[1 of 1] Compiling Main             ( snap-file-fail.hs, snap-file-fail.o )
*** Parser [Main]:
!!! Parser [Main]: finished in 0.42 milliseconds, allocated 0.564 megabytes
*** Renamer/typechecker [Main]:
!!! Renamer/typechecker [Main]: finished in 11.10 milliseconds, allocated 10.580 megabytes

snap-file-fail.hs:16:1: error:
    Failed to load interface for ‘Snap.Internal.Routing’
    it is a hidden module in the package ‘snap-core-1.0.3.0’
    it is a hidden module in the package ‘snap-core-1.0.2.1’
    Locations searched:
      Snap/Internal/Routing.hs
      Snap/Internal/Routing.lhs
      Snap/Internal/Routing.hsig
      Snap/Internal/Routing.lhsig
Upsweep partially successful.
*** Deleting temp files:
Deleting: 
link(batch): upsweep (partially) failed OR
   Main.main not exported; not linking.
*** Deleting temp files:
Deleting: 
*** Deleting temp dirs:
Deleting: 
Charless-MacBook-Pro-3:Threepenny-gui charlesschatz$ ghc snap-file-fail.hs 
[1 of 1] Compiling Main             ( snap-file-fail.hs, snap-file-fail.o )

snap-file-fail.hs:23:47: error:
    • No instance for (MonadSnap
                         snap-core-1.0.2.1:Snap.Internal.Core.Snap)
        arising from a use of ‘route’
      There are instances for similar types:
        instance MonadSnap Snap -- Defined in ‘Snap.Internal.Core’
    • In the second argument of ‘($)’, namely
        ‘route [("/hy.jpg", serveFile path)]’
      In a stmt of a 'do' block:
        httpServe (setPort 10000 defaultConfig)
        $ route [("/hy.jpg", serveFile path)]
      In the expression:
        do { let path
                   = "/Users/charlesschatz/Desktop/Classical Objects and Scenes/hy.jpg";
             putStrLn . ("File size is " ++) . show . fileSize
             =<< getFileStatus path;
             httpServe (setPort 10000 defaultConfig)
             $ route [("/hy.jpg", serveFile path)] }

The same error occurs with your original code.

I'm too tired to hunt down the correct instance for 'route' (I'm not a native Haskell programmer) -
I am researching other stuff while doing this. I seem to have two installed versions of snap-core: 1.0.2.1 and 1.0.3.0. Should I eliminate one of them?

[HeinrichApfelmus]

If you compile directly with ghc, the global package database is used. It appears that you have to versions of the package in the database, which is not necessarily a problem, but both are hidden. Try

ghc-pkg expose snap-core-1.0.3.0

You can inspect the global package database with

ghc-pkg list

Sorry that I can't help much on this. The Haskell IRC channel is a good place to get live help on installation issues.

--

Also note that specifying a MIME type for the files may help, in case you didn't do that already.

[ChasTheSpaz]

I performed the exposure of snap-core-1.0.3.0, no effect on the erroneous compile.
I tried to unregister snap-core-1.0.2.1 and got this result:

$ ghc-pkg unregister snap-core-1.0.2.1
ghc-pkg: unregistering would break the following packages: threepenny-gui-0.8.0.0 websockets-snap-0.10.2.2 snap-server-1.0.2.2 (use --force to override)

The above seems to imply that 3penny 8.0.0 AND other snap bits and pieces are compiled against 1.0.2,1. Looks like dependency Hell has entered the scene.

[ChasTheSpaz]

Finally got a SNAP example to compile. It is indeed a problem for the SNAP framework to deal with huge files. It completely fails to successfully serve my largest JPEG file of 13 megabytes and damages the display of somewhat smaller files. It appears to work well on much smaller files, but I can't control the file size of potential images.