Goofys is a Filey-System interface to S3
Goofys allows you to mount an S3 bucket as a filey system.
It's a Filey System instead of a File System because goofys strives
for performance first and POSIX second. Particularly things that are
difficult to support on S3 or would translate into more than one
round-trip would either fail (random writes) or faked (no per-file
permission). Goofys does not have a on disk data cache, and
consistency model is close-to-open.
Pre-built binaries are available here.
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys
$ cat > ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys <bucket> <mountpoint>Users can also configure credentials via the
AWS CLI
or the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables.
Using --stat-cache-ttl 0 --type-cache-ttl 0 for goofys
-ostat_cache_expire=1 for s3fs to simulate cold runs. Detail for the
benchmark can be found in
bench.sh. Raw data
is available as well. Test was run on an EC2 c4.xlarge in us-west-2a
connecting to a bucket in us-west-2.
| operation | goofys | s3fs | speedup |
|---|---|---|---|
| Create 1000 files | 49.4 +/- 1.5s | 146.0 +/- 15.0s | 2.96 +/- 0.32x |
| Unlink 1000 files | 28.1 +/- 0.8s | 36.7 +/- 6.2s | 1.31 +/- 0.22x |
| ls with 1000 files | 0.21 +/- 0.04s | 3.5 +/- 0.6s | 16.9 +/- 4.6x |
| Create 1000 files (parallel) | 21.5 +/- 0.4s | 134.2 +/- 9.1s | 6.2 +/- 0.4x |
| Unlink 1000 files (parallel) | 28.18 +/- 0.35s | 38.1 +/- 4.2s | 1.35 +/- 0.15x |
| Write 1GB | 51.4 +/- 4.3MB/s | 29.7 +/- 2.9MB/s | 1.73 +/- 0.22x |
| Read 1GB | 58.9 +/- 4.7MB/s | 65.7 +/- 18.9MB/s | 0.90 +/- 0.27x |
| Time to 1st byte | 0.0169 +/- 0.0023s | 0.98 +/- 0.06s | 58.3 +/- 8.7x |
Copyright (C) 2015 Ka-Hing Cheung
Licensed under the Apache License, Version 2.0
List of not yet implemented fuse operations:
- in terms of syscalls
readlinkchmod/utimes/ftruncate
- in terms of fuse functions
ReadSymlinkSetInodeAttributes
List of non-POSIX behaviors/limitations:
- only sequential writes supported
- does not support appending to a file yet
- file mode is always
0644for regular files and0700for directories - directories link count is always
2 - file owner is always the user running goofys
ctime,atimeis always the same asmtime- cannot rename non-empty directories
unlinkreturns success even if file is not present- can only create files up to 50GB
- Data is stored on Amazon S3
- Amazon SDK for Go
- Other related fuse filesystems
- s3fs: another popular filesystem for S3
- gcsfuse: filesystem for Google Cloud Storage. Goofys borrowed some skeleton code from this project.
- S3Proxy is used for
go test - fuse binding, also used by
gcsfuse