Document and provide examples how to do multipart/formdata uploads on the client side

[akka-ci]

Issue by jrudolph
Friday Jun 05, 2015 at 16:17 GMT
Originally opened as akka/akka#17665

---

Low-level and with marshalling.

/cc @sirthias

[akka-ci]

Comment by jrudolph
Monday Jun 08, 2015 at 09:59 GMT

---

Here's a basic example I hacked together: https://gist.github.com/jrudolph/08d0d28e1eddcd64dbd0

[akka-ci]

Comment by nykolaslima
Monday Jun 15, 2015 at 18:12 GMT

---

Hi @jrudolph.
I'm pretty new to akka but in my opinion this example that you hacked is too complex.
File upload should be a simple feature, it's made with just a few lines in the majority of the other frameworks. I believe that we should have a simple example, that shows how to receive the entity body and upload it to somewhere. We could also have more complex examples, but this basic one is much needed IMHO.

[akka-ci]

Comment by jrudolph
Tuesday Jun 16, 2015 at 08:11 GMT

---

@nykolaslima thanks for the comment. We are always looking for suggestions how to simplify things. Can you give an example about how you would expect it to look like (or how other client libraries do this)? The part of creating a file upload request isn't not as short as it could be but the 10+ lines don't seem too bad:

  def createFileUploadEntityFromFile(file: File): Future[RequestEntity] = {
    require(file.exists())
    val formData =
      Multipart.FormData(
        Source.single(
          Multipart.FormData.BodyPart(
            "test",
            HttpEntity(MediaTypes.`application/octet-stream`, file.length(), SynchronousFileSource(file, chunkSize = 100000)), // the chunk size here is currently critical for performance
            Map("filename" -> file.getName))))
    Marshal(formData).to[RequestEntity]
  }

  def createRequest(target: Uri, file: File): Future[HttpRequest] =
    for {
      e ← createFileUploadEntityFromFile(file)
    } yield HttpRequest(HttpMethods.POST, uri = target, entity = e)

I guess it would be nice if there were

• a FormData.BodyPart constructor that takes a file as an argument
• a FormData constructor that takes one or more files as an argument
• a FormData.toRequestEntity method that would allow marshalling directly from the model

This example could then be written as

  def createFileUploadEntityFromFile(file: File): Future[RequestEntity] = {
    require(file.exists())
    Multipart.FormData.fromFile(file, chunkSize = ...).toRequestEntity
  }

  def createRequest(target: Uri, file: File): Future[HttpRequest] =
    for {
      e ← createFileUploadEntityFromFile(file)
    } yield HttpRequest(HttpMethods.POST, uri = target, entity = e)

[akka-ci]

Comment by nykolaslima
Wednesday Jun 17, 2015 at 12:55 GMT

---

Actually I got a really hard day trying to make file upload to work hahaha

I got a solution and I think it's pretty easy and would be a nice documentation example

(post & path("photos")) {
      entity(as[Multipart.FormData]) { (formData) =>

        val uploadedUrlsFuture = formData.parts.map(_.entity.dataBytes).mapAsync(parallelism = 1)(part =>
          part
            .map(_.toArray)
            .runFold(Array[Byte]())((totalBytes, bytes) => totalBytes ++ bytes)
            .map(photosService.upload(_))
        ).grouped(1000).runWith(Sink.head)

        complete(OK)
      }
    }

What do you think? Maybe removing the _ and using the types variables would make it more understandable for the documentation.

[akka-ci]

Comment by jrudolph
Wednesday Jun 17, 2015 at 13:36 GMT

---

@nykolaslima ah, you are talking about the server side. This issue originally was about the client-side and I implemented the server side just for testing purposes. Still thanks for the suggestions :). There's also #16841 to improve accessing uploaded files on the server side.

Regarding the code itself, there are at least these problems:

• you are collecting into an ever growing array, this has quadratic runtime because you are copying more and more data for every new chunks that arrives
• you are limited to 2^31 bytes
• you are collecting data in the heap, the very problem akka-http and akka-stream are trying to prevent

What you should do instead is passing the source of the part (part.entity.dataBytes) to your backend service and keep it as along as possible and connect it only at the end to something which can consume things in a streaming manner. An obvious choice for file uploads would be a SynchronousFileSink to write data to disk.

[akka-ci]

Comment by nykolaslima
Wednesday Jun 17, 2015 at 13:54 GMT

---

@jrudolph thank you for the feedback!

I don't understand how to do it. The photosService should receive a stream? But the photosService will only be able to do something when all the bytes have arrived.

[akka-ci]

Comment by drewhk
Tuesday Aug 25, 2015 at 12:54 GMT

---

@nykolaslima That is because the photosService.upload() function is not streaming but needs all the data to be drained - which is suboptimal and defeats the whole benefit of streaming. Sometimes you cannot avoid this because of a 3rd party library, but in many cases you can just stream the bytes directly instead of buffering them to the heap.

And yes, ++ on the incoming Array chunks will be quadratic. Use the ++ from ByteString instead and get the underlying Array as the last step instead (if you really need the Array).

[akka-ci]

Comment by clockfly
Thursday Sep 10, 2015 at 09:05 GMT

---

To those who may be interested about how to create a DSL directive to do this:

usage:

  val route: Route = {
    path("upload") {
      uploadFile { fileMap =>
        complete(ToResponseMarshallable(fileMap))
      }
    }
}

example code:
https://github.com/clockfly/akka-http-file-server

[akka-ci]

Comment by jrudolph
Thursday Sep 10, 2015 at 09:25 GMT

---

@clockfly thanks for sharing!

[akka-ci]

Comment by 2beaucoup
Thursday Sep 10, 2015 at 14:06 GMT

---

I think that a

def uploadToDirectory(dir: File, maxFiles: Int = 1): Directive0

and/or

def upload(maxFiles: Int = 1): Directive1[Source[(FileInfo, Source[ByteString])]]

could be a nice addition to the FileAndResourceDirectives.

[2m]

One more java example for the reference: 2m@6562550

[ihrimech]

And for the reference, an another example on how to use Multipart.FormData on the client side :

// 1- reading a file
val source = new File(
        getClass.getResource("/example.pdf").getFile
      )

// 2- a method to construct entities
def defaultEntity(content: String) =
        HttpEntity.Strict(ContentTypes.`text/plain(UTF-8)`, ByteString(content))

// 3- then the multipartForm 
val multipartForm = Multipart.FormData(Source(
        Multipart.FormData.BodyPart("someKey", defaultEntity("SomeValue")) ::
          Multipart.FormData.BodyPart("AnotherKey", defaultEntity("AnotherValue")) ::
          Multipart.FormData.BodyPart.fromFile(
            "file", ContentType.Binary(MediaTypes.`application/pdf`), source
          ):: Nil))

4- You can then construct the request as @jrudolph mentionned :

def createRequest(target: Uri, file: File): Future[HttpRequest] =
    for {
      e ← createFileUploadEntityFromFile(file)
    } yield HttpRequest(HttpMethods.POST, uri = target, entity = e)