awesome-very-deep-learning
awesome-very-deep-learning is a curated list for papers and code about implementing and training very deep neural networks.
Deep Residual Learning
Deep Residual Networks are a family of extremely deep architectures (up to 1000 layers) showing compelling accuracy and nice convergence behaviors. Instead of learning a new representation at each layer, deep residual networks use identity mappings to learn residuals.
Papers
- Wide Residual Networks (2016) [orginal code], studies wide residual neural networks and shows that making residual blocks wider outperforms deeper and thinner network architectures
- Swapout: Learning an ensemble of deep architectures (2016), improving accuracy by randomly applying dropout, skipforward and residual units per layer
- Deep Networks with Stochastic Depth (2016) [original code], dropout with residual layers as regularizer
- Identity Mappings in Deep Residual Networks (2016) [original code], improving the original proposed residual units by reordering batchnorm and activation layers
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning (2016), inception network with residual connections
- Deep Residual Learning for Image Recognition (2015) [original code], original paper introducing residual neural networks
Implementations
- Torch by Facebook AI Research (FAIR), with training code in Torch and pre-trained ResNet-18/34/50/101 models for ImageNet: blog, code
- Torch, CIFAR-10, with ResNet-20 to ResNet-110, training code, and curves: code
- Lasagne, CIFAR-10, with ResNet-32 and ResNet-56 and training code: code
- Neon, CIFAR-10, with pre-trained ResNet-32 to ResNet-110 models, training code, and curves: code
- Neon, Preactivation layer implementation: code
- Torch, MNIST, 100 layers: blog, code
- A winning entry in Kaggle's right whale recognition challenge: blog, code
- Neon, Place2 (mini), 40 layers: blog, code
- Tensorflow with tflearn, with CIFAR-10 and MNIST: code
- Tensorflow with skflow, with MNIST: code
- Stochastic dropout in Keras: code
- ResNet in Chainer: code
- Stochastic dropout in Chainer: code
- Wide Residual Networks in Keras: code
- ResNet in TensorFlow 0.9+ with pretrained caffe weights: code
In addition, this code by Ryan Dahl helps to convert the pre-trained models to TensorFlow.
Highway Networks
Highway Networks take inspiration from Long Short Term Memory (LSTM) and allow training of deep, efficient networks (with hundreds of layers) with conventional gradient-based methods
Papers
- Training Very Deep Networks (2015), introducing highway neural networks
Implementations
Very Deep Learning Theory
Theories in very deep learning concentrate on the ideas that very deep networks with skip connections are able to efficiently approximate recurrent computations (similar to the recurrent connections in the visual cortex) or are actually exponential ensembles of shallow networks
Papers
- Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex, shows that ResNets with shared weights work well too although having fewer parameters
- Residual Networks are Exponential Ensembles of Relatively Shallow Networks, shows that ResNets behaves just like ensembles of shallow networks in test time. This suggests that in addition to describing neural networks in terms of width and depth, there is a third dimension: multiplicity, the size of the implicit ensemble
最新文章
- Web前端开发高手进阶
- CSS中如何实现未知尺寸图片垂直居中
- 跟着鸟哥学Linux系列笔记0-如何解决问题
- 流镜像（华为S9306和S5700）
- iOS获取设备信息
- Android 学习笔记 Service服务与远程通信...（AIDL）
- PDF数据提取------3.解析Demo
- JS操作JSON总结(转)
- java+android学习路线图
- java分布式服务框架Dubbo的介绍与使用
- Mac下eclipse导入其他工程中文注释出现乱码解决方案
- JS题目
- 产品设计-后台管理权限设计RBAC
- 手机端input获取焦点弹出键盘时挡住input解决方案
- Mysql半同步复制模式说明及配置示例 - 运维小结
- [daily][archlinux][pacman] 删除所有孤立包(orphan)
- 《汇编语言 基于x86处理器》第八章高级过程部分的代码 - 两种规范计算数组元素的和
- li之间的间隙问题
- 开发只懂 AFN ？搞定 NSURLSession 才是硬道理
- 线性DP总结（LIS,LCS,LCIS，最长子段和）