Goofys is a Filey-System interface to S3
Goofys allows you to mount an S3 bucket as a filey system.
It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have a on disk data cache, and consistency model is close-to-open.
Pre-built binaries are available here. You may also need to install fuse-utils first.
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys
$ cat > ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys <bucket> <mountpoint>Users can also configure credentials via the
AWS CLI
or the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables.
Using --stat-cache-ttl 0 --type-cache-ttl 0 for goofys
-ostat_cache_expire=1 for s3fs to simulate cold runs. Detail for the
benchmark can be found in
bench.sh. Raw data
is available as well. Test was run on an EC2 c4.xlarge in us-west-2a
connecting to a bucket in us-west-2. Units are seconds.
| operation | goofys | s3fs | speedup over s3fs | riofs† | speedup over riofs |
|---|---|---|---|---|---|
| Create 100 files | 5.31+/-0.35 | 33.7+/-2.5 | 6.3+/-0.6x | 1.43+/-0.21† | 0.27+/-0.04x |
| Unlink 100 files | 3.1+/-0.4* | 6.6+/-0.6 | 2.12+/-0.32x | 3.63+/-0.33 | 1.17+/-0.17x |
| Create 100 files (parallel) | 2.45+/-0.30 | 29.4+/-1.7* | 12.0+/-1.6x | 1.25+/-0.16† | 0.51+/-0.09x |
| Unlink 100 files (parallel) | 3.10+/-0.35 | 9.7+/-1.7 | 3.1+/-0.7x | 4.2+/-0.4 | 1.36+/-0.20x |
| ls with 1000 files | 0.173+/-0.029* | 9.6+/-1.4 | 55.2+/-12.3x | 0.21+/-0.09* | 1.2+/-0.6x |
| Write 1GB | 11.7+/-1.8* | 38.4+/-6.2* | 3.3+/-0.7x | 117.1+/-3.7 | 10.0+/-1.6x |
| Read 1GB | 17.1+/-1.1* | 22.0+/-6.7* | 1.3+/-0.4x | 25.2+/-1.0 | 1.48+/-0.11x |
| Time to 1st byte | 0.036+/-0.013* | 1.1+/-0.4 | 31.0+/-16.8x | 0.275+/-0.018* | 7.6+/-2.9x |
(*) indicates the number of outliers removed
(†) riofs does not wait for HTTP response before returning from release()
Copyright (C) 2015 Ka-Hing Cheung
Licensed under the Apache License, Version 2.0
List of not yet implemented fuse operations:
- in terms of syscalls
readlinkchmod/utimes/ftruncate
- in terms of fuse functions
ReadSymlinkSetInodeAttributes
List of non-POSIX behaviors/limitations:
- only sequential writes supported
- does not support appending to a file yet
- file mode is always 0644 for regular files and 0700 for directories
- directories link count is always 2
- file owner is always the user running goofys
ctime,atimeis always the same asmtime- cannot rename non-empty directories
unlinkreturns success even if file is not present- can only create files up to 50GB
- Data is stored on Amazon S3
- Amazon SDK for Go
- Other related fuse filesystems
- s3fs: another popular filesystem for S3
- gcsfuse: filesystem for Google Cloud Storage. Goofys borrowed some skeleton code from this project.
- S3Proxy is used for
go test - fuse binding, also used by
gcsfuse